[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:09 dist-test-slave-92t4 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:09 dist-test-slave-92t4 krb5kdc[399](info): set up 2 sockets
Jun 20 09:05:09 dist-test-slave-92t4 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 09:05:10.910318 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:10 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:11.032924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:11.145583 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:11.145932 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:11.146008 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:11.150612 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:11.150695 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:11.150732 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:11.150774 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:11.150811 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:11.156370 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:11.157565 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:11.158699 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:11.167929 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.168056 412 server_base.cc:1034] running on GCE node
W20260620 09:05:11.167982 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.167968 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.168926 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:11.170194 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:11.171386 412 hybrid_clock.cc:648] HybridClock initialized: now 1781946311171343 us; error 58 us; skew 500 ppm
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:11.174336 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:11.175180 412 webserver.cc:460] Webserver started at http://127.0.98.62:38901/ using document root <none> and password file <none>
I20260620 09:05:11.175715 412 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:11.175771 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:11.175963 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:11.178089 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "d7cd2b2676b747d89dc634edd298c063"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c6c24e7f0c267f20292f59fe1f055887"
server_key_iv: "d947ce338474e3880ac724af7deddd30"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.178604 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7cd2b2676b747d89dc634edd298c063"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c6c24e7f0c267f20292f59fe1f055887"
server_key_iv: "d947ce338474e3880ac724af7deddd30"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.182667 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 09:05:11.185236 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.186471 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:11.186602 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "d7cd2b2676b747d89dc634edd298c063"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c6c24e7f0c267f20292f59fe1f055887"
server_key_iv: "d947ce338474e3880ac724af7deddd30"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.186712 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:11.203464 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:11.206568 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:11.206840 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:11.214193 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46611
I20260620 09:05:11.214211 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46611 every 8 connection(s)
I20260620 09:05:11.215185 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:11.218154 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:11.219978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 412
I20260620 09:05:11.220157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:11.221837 392 external_mini_cluster.cc:1428] Setting key ece86455260c550a030573d4352f72ad
I20260620 09:05:11.225322 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063: Bootstrap starting.
I20260620 09:05:11.227908 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:11.228709 480 log.cc:824] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:11.230831 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063: No bootstrap required, opened a new log
I20260620 09:05:11.234930 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } }
I20260620 09:05:11.235206 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:11.235256 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cd2b2676b747d89dc634edd298c063, State: Initialized, Role: FOLLOWER
I20260620 09:05:11.235738 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } }
I20260620 09:05:11.235939 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:11.236012 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:11.236150 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:11.237154 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } }
I20260620 09:05:11.237511 480 leader_election.cc:302] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7cd2b2676b747d89dc634edd298c063; no voters:
I20260620 09:05:11.237875 480 leader_election.cc:288] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:11.237988 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:11.238277 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [term 1 LEADER]: Becoming Leader. State: Replica: d7cd2b2676b747d89dc634edd298c063, State: Running, Role: LEADER
I20260620 09:05:11.238658 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } }
I20260620 09:05:11.239183 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:11.240374 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7cd2b2676b747d89dc634edd298c063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } } }
I20260620 09:05:11.240513 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:11.240839 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7cd2b2676b747d89dc634edd298c063. Latest consensus state: current_term: 1 leader_uuid: "d7cd2b2676b747d89dc634edd298c063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cd2b2676b747d89dc634edd298c063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46611 } } }
I20260620 09:05:11.240921 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:11.241597 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:11.244762 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:11.248498 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.225661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.98.62:46611)
0620 09:05:11.226221 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:11.226241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:11.227815 (+ 1574us) server_negotiation.cc:407] Connection header received
0620 09:05:11.231249 (+ 3434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:11.231283 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:11.231683 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:11.232040 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:11.234238 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.235741 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.237161 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.237847 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.238022 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.242950 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:11.242955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:11.242968 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:11.242993 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:11.244794 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.245433 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.245438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.245442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.245549 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.245915 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.245922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.245924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.246294 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:11.246506 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:11.247174 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 09:05:11.247487 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":188,"threads_started":1}
I20260620 09:05:11.250931 494 catalog_manager.cc:1347] Generated new cluster ID: c9f1b593f5b94daf84bfb92c114f677b
I20260620 09:05:11.250988 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:11.275885 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:11.277247 494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:11.291582 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7cd2b2676b747d89dc634edd298c063: Generated new TSK 0
I20260620 09:05:11.292497 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:11.323880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:11.438573 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:11.438930 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:11.439041 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:11.443451 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:11.443526 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:11.443630 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:11.449124 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:11.450300 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:11.451385 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:11.461309 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.461375 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.461342 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.461573 506 server_base.cc:1034] running on GCE node
I20260620 09:05:11.462141 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:11.462793 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:11.463982 506 hybrid_clock.cc:648] HybridClock initialized: now 1781946311463965 us; error 31 us; skew 500 ppm
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:11.466790 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:11.467730 506 webserver.cc:460] Webserver started at http://127.0.98.1:35189/ using document root <none> and password file <none>
I20260620 09:05:11.468310 506 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:11.468403 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:11.468633 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:11.470813 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ca28bbbd35f4e7590d7f3b585c1111d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "0e997a4ce0dcd4da619c83edd821886a"
server_key_iv: "21c13c0ae1ce0768a68364b3b5516366"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.471402 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ca28bbbd35f4e7590d7f3b585c1111d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "0e997a4ce0dcd4da619c83edd821886a"
server_key_iv: "21c13c0ae1ce0768a68364b3b5516366"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.475241 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 09:05:11.477702 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.478987 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:11.479115 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "5ca28bbbd35f4e7590d7f3b585c1111d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "0e997a4ce0dcd4da619c83edd821886a"
server_key_iv: "21c13c0ae1ce0768a68364b3b5516366"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.479229 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:11.496580 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:11.500721 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:11.500943 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:11.501583 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:11.502454 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:11.502539 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.502607 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:11.502657 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.511750 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44925
I20260620 09:05:11.511771 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44925 every 8 connection(s)
I20260620 09:05:11.512688 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:11.520581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 506
I20260620 09:05:11.520718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:11.521011 392 external_mini_cluster.cc:1428] Setting key 24b35066caf6fef04bb6a9c7f20ba240
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:11.527055 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.514635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55501 (local address 127.0.98.62:46611)
0620 09:05:11.514770 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:11.514774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:11.515644 (+ 870us) server_negotiation.cc:407] Connection header received
0620 09:05:11.517075 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:11.517079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:11.517139 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:11.517213 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:11.518395 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.518937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.520084 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.520362 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.520479 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.523183 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:11.523191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:11.523194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:11.523216 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:11.524698 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.525387 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.525390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.525391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.525474 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.525869 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.525871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.525872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.526044 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:11.526123 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:11.526812 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 09:05:11.526929 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:11.528498 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.515048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46611 (local address 127.0.98.1:55501)
0620 09:05:11.515486 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:11.515541 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:11.516828 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:11.517388 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:11.517395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:11.517761 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:11.518247 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.518260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.519068 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.519073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.519958 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.519967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.520515 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.520519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.520691 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.521393 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:11.521409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:11.523019 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:11.524839 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.524846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.524861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.525271 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.525567 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.525573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.525578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.525776 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.526148 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:11.526158 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:11.526540 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 09:05:11.526786 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 09:05:11.527182 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260620 09:05:11.529844 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46611
I20260620 09:05:11.530129 635 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:11.530790 635 heartbeater.cc:511] Master 127.0.98.62:46611 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:11.532506 444 ts_manager.cc:194] Registered new tserver with Master: 5ca28bbbd35f4e7590d7f3b585c1111d (127.0.98.1:44925)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:05:11.534017 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:11.548372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:11.661298 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:11.661600 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:11.661672 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:11.665766 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:11.665854 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:11.665943 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:11.671353 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:11.672508 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:11.673627 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:11.683735 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.683728 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.683863 640 server_base.cc:1034] running on GCE node
W20260620 09:05:11.684087 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.684576 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:11.685334 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:11.686532 640 hybrid_clock.cc:648] HybridClock initialized: now 1781946311686491 us; error 51 us; skew 500 ppm
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:11.689432 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:11.690408 640 webserver.cc:460] Webserver started at http://127.0.98.2:35029/ using document root <none> and password file <none>
I20260620 09:05:11.691000 640 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:11.691066 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:11.691241 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:11.693498 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "b38bf3b1d07a4f8a8d5db2fdd7309a1d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "948587c632561e55d576d562b1d14361"
server_key_iv: "5d58e6571e4955cb8ad0e438a4c09f1d"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.694348 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b38bf3b1d07a4f8a8d5db2fdd7309a1d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "948587c632561e55d576d562b1d14361"
server_key_iv: "5d58e6571e4955cb8ad0e438a4c09f1d"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.698236 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:05:11.700601 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.701915 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:11.702013 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "b38bf3b1d07a4f8a8d5db2fdd7309a1d"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "948587c632561e55d576d562b1d14361"
server_key_iv: "5d58e6571e4955cb8ad0e438a4c09f1d"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.702098 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:11.712430 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:11.715844 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:11.716002 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:11.716518 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:11.717363 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:11.717418 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.717459 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:11.717480 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.727846 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34195
I20260620 09:05:11.727881 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34195 every 8 connection(s)
I20260620 09:05:11.728801 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:11.735009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 640
I20260620 09:05:11.735144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:11.735437 392 external_mini_cluster.cc:1428] Setting key beafadec187c347fff5cff489bfb694b
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:11.743896 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.730715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56921 (local address 127.0.98.62:46611)
0620 09:05:11.730857 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:11.730860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:11.731683 (+ 823us) server_negotiation.cc:407] Connection header received
0620 09:05:11.733027 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:11.733031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:11.733107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:11.733215 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:11.734378 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.735022 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.736326 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.736646 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.736777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.739969 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:11.739973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:11.739976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:11.740011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:11.741698 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.742332 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.742336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.742338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.742430 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.742793 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.742796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.742798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.742988 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:11.743094 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:11.743608 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 09:05:11.743730 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:05:11.744771 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.731108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46611 (local address 127.0.98.2:56921)
0620 09:05:11.731532 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:11.731574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:11.732814 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:11.733391 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:11.733398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:11.733747 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:11.734223 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.734236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.735167 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.735174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.736198 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.736209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.736832 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.736835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.737039 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.737704 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:11.737732 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:11.739712 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:11.741827 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.741835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.741851 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.742223 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.742529 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.742533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.742535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.742678 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.743100 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:11.743109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:11.743368 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 09:05:11.743558 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 09:05:11.743921 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:11.746074 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46611
I20260620 09:05:11.746335 769 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:11.746871 769 heartbeater.cc:511] Master 127.0.98.62:46611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:05:11.747953 444 ts_manager.cc:194] Registered new tserver with Master: b38bf3b1d07a4f8a8d5db2fdd7309a1d (127.0.98.2:34195)
I20260620 09:05:11.748729 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:11.763592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:11.876739 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:11.877111 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:11.877205 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:11.881390 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:11.881476 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:11.881567 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:11.886909 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:11.888041 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:11.889158 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:11.898729 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.898715 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:11.898710 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:11.899190 774 server_base.cc:1034] running on GCE node
I20260620 09:05:11.899757 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:11.900386 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:11.901582 774 hybrid_clock.cc:648] HybridClock initialized: now 1781946311901563 us; error 30 us; skew 500 ppm
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:11.904428 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:11.905454 774 webserver.cc:460] Webserver started at http://127.0.98.3:33677/ using document root <none> and password file <none>
I20260620 09:05:11.906038 774 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:11.906128 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:11.906388 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:11.908474 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "c78db582507d4c37a2c30fa2a42ec7f7"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c44055c46da04f068ab0c761f6714a15"
server_key_iv: "57ffa9d9e09c1dc412bedfdc3dafbde0"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.909106 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c78db582507d4c37a2c30fa2a42ec7f7"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c44055c46da04f068ab0c761f6714a15"
server_key_iv: "57ffa9d9e09c1dc412bedfdc3dafbde0"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.913103 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:05:11.916047 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.917433 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:11.917574 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "c78db582507d4c37a2c30fa2a42ec7f7"
format_stamp: "Formatted at 2026-06-20 09:05:11 on dist-test-slave-92t4"
server_key: "c44055c46da04f068ab0c761f6714a15"
server_key_iv: "57ffa9d9e09c1dc412bedfdc3dafbde0"
server_key_version: "encryptionkey@0"
I20260620 09:05:11.917696 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:11.928696 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:11.931695 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:11.931915 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:11.932526 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:11.933429 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:11.933496 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.933563 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:11.933614 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:11.943868 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42065
I20260620 09:05:11.943897 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42065 every 8 connection(s)
I20260620 09:05:11.944808 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:11.949955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 774
I20260620 09:05:11.950079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:11.950380 392 external_mini_cluster.cc:1428] Setting key ee6a7fee478a652ca09aed4bdc5b603f
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:11.958848 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.946696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44481 (local address 127.0.98.62:46611)
0620 09:05:11.946845 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:11.946848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:11.947684 (+ 836us) server_negotiation.cc:407] Connection header received
0620 09:05:11.948982 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:11.948985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:11.949081 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:11.949172 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:11.950570 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.951208 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.952770 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.952986 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.953143 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.955530 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:11.955533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:11.955537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:11.955567 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:11.956768 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.957362 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.957366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.957368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.957447 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.957779 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.957784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.957786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.957966 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:11.958070 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:11.958563 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 09:05:11.958677 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 09:05:11.959684 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.947088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46611 (local address 127.0.98.3:44481)
0620 09:05:11.947517 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:11.947559 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:11.948787 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:11.949321 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:11.949328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:11.949719 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:11.950417 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.950436 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.951330 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.951333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.952655 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:11.952666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.953191 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.953195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:11.953370 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.953934 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:11.953949 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:11.955369 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:11.956893 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.956899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.956914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.957242 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.957541 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:11.957546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:11.957548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:11.957686 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:11.958078 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:11.958087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:11.958335 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 09:05:11.958523 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 09:05:11.958845 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260620 09:05:11.960759 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46611
I20260620 09:05:11.961004 903 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:11.961501 903 heartbeater.cc:511] Master 127.0.98.62:46611 requested a full tablet report, sending...
I20260620 09:05:11.962491 444 ts_manager.cc:194] Registered new tserver with Master: c78db582507d4c37a2c30fa2a42ec7f7 (127.0.98.3:42065)
I20260620 09:05:11.963063 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44481
I20260620 09:05:11.964697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:11.990775 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:11.981012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.98.1:44925)
0620 09:05:11.981386 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:11.981400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:11.981425 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:05:11.981480 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:11.981486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:11.981682 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:11.981803 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:11.982491 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.983332 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.984263 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:11.984639 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:11.984718 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:11.987167 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:11.987175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:11.987190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:11.987215 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:11.989023 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.989508 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.989513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.989514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.989593 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:11.989824 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:11.989826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:11.989827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:11.990067 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:11.990234 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:11.990395 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:05:11.990590 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
W20260620 09:05:11.991824 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59210
Jun 20 09:05:11 dist-test-slave-92t4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:05:12 dist-test-slave-92t4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:12.017886 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:12.009495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.0.98.1:44925)
0620 09:05:12.009645 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:12.009650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:12.009800 (+ 150us) server_negotiation.cc:407] Connection header received
0620 09:05:12.009917 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:12.009920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:12.009963 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:12.010068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:12.010745 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:12.011255 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:12.012059 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:12.012274 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:12.012355 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:12.014871 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:12.014874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:12.014876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:12.014900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:12.016383 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:12.016749 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:12.016751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:12.016752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:12.016824 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:12.017102 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:12.017108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:12.017110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:12.017279 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:12.017385 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:12.017585 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:05:12.017683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:05:12.018905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 506
I20260620 09:05:12.024982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 640
I20260620 09:05:12.032866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 774
I20260620 09:05:12.038434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 412
2026-06-20T09:05:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2188 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:12 dist-test-slave-92t4 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:12 dist-test-slave-92t4 krb5kdc[917](info): set up 2 sockets
Jun 20 09:05:12 dist-test-slave-92t4 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 09:05:13.080667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:13 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 09:05:13.750269 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.573s user 0.001s sys 0.002s
I20260620 09:05:13.750828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:13.861881 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:13.862159 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:13.862197 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:13.866407 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:13.866474 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:13.866494 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:13.866511 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:13.866525 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:13.871906 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:13.873006 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:13.874039 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:13.882643 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:13.882645 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:13.882794 930 server_base.cc:1034] running on GCE node
W20260620 09:05:13.882642 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:13.883549 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:13.884687 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:13.885883 930 hybrid_clock.cc:648] HybridClock initialized: now 1781946313885862 us; error 30 us; skew 500 ppm
Jun 20 09:05:13 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:13.888597 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:13.889556 930 webserver.cc:460] Webserver started at http://127.0.98.62:45041/ using document root <none> and password file <none>
I20260620 09:05:13.890084 930 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:13.890141 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:13.890316 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:13.892305 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "f8fab451055547bc99b183a32e96fe38"
format_stamp: "Formatted at 2026-06-20 09:05:13 on dist-test-slave-92t4"
server_key: "03a86c90f652968364527c27c9dd194f"
server_key_iv: "fb977d66798b4a056944f079c1da12a7"
server_key_version: "encryptionkey@0"
I20260620 09:05:13.892803 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8fab451055547bc99b183a32e96fe38"
format_stamp: "Formatted at 2026-06-20 09:05:13 on dist-test-slave-92t4"
server_key: "03a86c90f652968364527c27c9dd194f"
server_key_iv: "fb977d66798b4a056944f079c1da12a7"
server_key_version: "encryptionkey@0"
I20260620 09:05:13.896482 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:13.898923 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:13.900147 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:13.900288 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "f8fab451055547bc99b183a32e96fe38"
format_stamp: "Formatted at 2026-06-20 09:05:13 on dist-test-slave-92t4"
server_key: "03a86c90f652968364527c27c9dd194f"
server_key_iv: "fb977d66798b4a056944f079c1da12a7"
server_key_version: "encryptionkey@0"
I20260620 09:05:13.900395 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:13.910056 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:13.913482 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:13.913712 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:13.920665 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42833
I20260620 09:05:13.920717 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42833 every 8 connection(s)
I20260620 09:05:13.921730 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:13.924515 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:13.927214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 930
I20260620 09:05:13.927310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:13.927565 392 external_mini_cluster.cc:1428] Setting key 298246badc78bca94e78560de3f73365
I20260620 09:05:13.931025 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: Bootstrap starting.
I20260620 09:05:13.933833 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:13.934697 998 log.cc:824] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:13.936921 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: No bootstrap required, opened a new log
Jun 20 09:05:13 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:13.940588 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 09:05:13.940902 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:13.940977 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fab451055547bc99b183a32e96fe38, State: Initialized, Role: FOLLOWER
I20260620 09:05:13.941504 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 09:05:13.941664 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:13.941733 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:13.941838 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:13.942823 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 09:05:13.943182 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:13.929096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.0.98.62:42833)
0620 09:05:13.929553 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:13.929574 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:13.929620 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:05:13.930451 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:13.930475 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:13.930899 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:13.931288 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:13.931960 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:13.933454 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:13.934365 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:13.935071 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:13.935225 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:13.938016 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:13.938021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:13.938035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:13.938074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:13.940180 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:13.940667 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:13.940671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:13.940676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:13.940800 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:13.941137 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:13.941140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:13.941142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:13.941543 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:13.941767 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:13.942051 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 09:05:13.942365 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260620 09:05:13.943169 998 leader_election.cc:302] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8fab451055547bc99b183a32e96fe38; no voters:
I20260620 09:05:13.943602 998 leader_election.cc:288] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:13.943673 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:13.943881 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [term 1 LEADER]: Becoming Leader. State: Replica: f8fab451055547bc99b183a32e96fe38, State: Running, Role: LEADER
I20260620 09:05:13.944180 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } }
I20260620 09:05:13.944928 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:13.947191 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8fab451055547bc99b183a32e96fe38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } } }
I20260620 09:05:13.947250 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fab451055547bc99b183a32e96fe38. Latest consensus state: current_term: 1 leader_uuid: "f8fab451055547bc99b183a32e96fe38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fab451055547bc99b183a32e96fe38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42833 } } }
I20260620 09:05:13.947589 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:13.947587 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38 [sys.catalog]: This master's current role is: LEADER
W20260620 09:05:13.949959 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:13.950034 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:13.950089 1018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:13.950919 1018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:13.956281 1018 catalog_manager.cc:1347] Generated new cluster ID: 42be0d0d399d461bb1fb7e8149abdaeb
I20260620 09:05:13.956359 1018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:13.981751 1018 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:13.983081 1018 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:13.992900 1018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8fab451055547bc99b183a32e96fe38: Generated new TSK 0
I20260620 09:05:13.993825 1018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:14.023767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:14.143626 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:14.143932 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:14.144012 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:14.148481 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:14.148552 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:14.148666 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:14.154220 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:14.155359 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:14.156430 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:14.166618 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:14.166608 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:14.166668 1024 server_base.cc:1034] running on GCE node
W20260620 09:05:14.166627 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:14.167279 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:14.167975 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:14.169117 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781946314169095 us; error 37 us; skew 500 ppm
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:14.171867 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:14.172757 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35333/ using document root <none> and password file <none>
I20260620 09:05:14.173408 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:14.173501 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:14.173734 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:14.175851 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "cdebd03d89d64d54b4c467750db73c08"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "ec4abd3863e260b28e8169455c43ae64"
server_key_iv: "6aa5cd3e3a53a9577eceb4003114b35d"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.176456 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cdebd03d89d64d54b4c467750db73c08"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "ec4abd3863e260b28e8169455c43ae64"
server_key_iv: "6aa5cd3e3a53a9577eceb4003114b35d"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.180317 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:05:14.182857 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.184167 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:14.184311 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "cdebd03d89d64d54b4c467750db73c08"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "ec4abd3863e260b28e8169455c43ae64"
server_key_iv: "6aa5cd3e3a53a9577eceb4003114b35d"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.184427 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:14.207337 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:14.210520 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:14.210738 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:14.211380 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:14.212252 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:14.212324 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.212399 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:14.212450 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.221279 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46203
I20260620 09:05:14.221299 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46203 every 8 connection(s)
I20260620 09:05:14.222261 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:14.230880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1024
I20260620 09:05:14.231005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:14.231316 392 external_mini_cluster.cc:1428] Setting key c660971249c84a98a4ab436f7669844e
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:14.237226 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.224116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33091 (local address 127.0.98.62:42833)
0620 09:05:14.224260 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.224263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.225249 (+ 986us) server_negotiation.cc:407] Connection header received
0620 09:05:14.226664 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.226668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.226739 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.226820 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.228068 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.228630 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.229849 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.230095 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.230210 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.233197 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.233200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.233202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.233228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.234733 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.235390 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.235393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.235394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.235466 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.235865 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.235871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.235876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.236139 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.236242 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.236775 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.236946 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:14.238050 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.224466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.1:33091)
0620 09:05:14.224967 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.225014 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.226433 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.227003 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.227011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.227395 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:14.227910 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.227924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.228772 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.228776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.229710 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.229719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.230256 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.230260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.230437 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.231216 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:14.231234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:14.233007 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:14.234857 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.234864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.234884 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.235284 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.235555 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.235557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.235558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.235745 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.236245 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:14.236250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:14.236512 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.236672 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.237103 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":122,"threads_started":1}
I20260620 09:05:14.239315 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 09:05:14.239579 1153 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:14.240082 1153 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:14.241657 962 ts_manager.cc:194] Registered new tserver with Master: cdebd03d89d64d54b4c467750db73c08 (127.0.98.1:46203)
I20260620 09:05:14.242916 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:14.259522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:14.375901 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:14.376228 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:14.376317 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:14.380573 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:14.380697 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:14.380815 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:14.386509 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:14.387802 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:14.388972 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:14.399464 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:14.399519 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:14.399574 1158 server_base.cc:1034] running on GCE node
W20260620 09:05:14.399464 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:14.400215 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:14.400947 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:14.402150 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781946314402131 us; error 32 us; skew 500 ppm
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:14.405110 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:14.406052 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45787/ using document root <none> and password file <none>
I20260620 09:05:14.406649 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:14.406762 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:14.407011 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:14.409291 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "055386aa4fe74abea05107fe92ed09be"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "230cb1d67215adc7c0db13ed26186ecb"
server_key_iv: "51d965d33b890158f5db2a236fe229fc"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.409893 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "055386aa4fe74abea05107fe92ed09be"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "230cb1d67215adc7c0db13ed26186ecb"
server_key_iv: "51d965d33b890158f5db2a236fe229fc"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.414027 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:14.416899 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.418352 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:05:14.418454 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "055386aa4fe74abea05107fe92ed09be"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "230cb1d67215adc7c0db13ed26186ecb"
server_key_iv: "51d965d33b890158f5db2a236fe229fc"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.418550 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:14.438970 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:14.442008 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:14.442167 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:14.442875 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:14.443706 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:14.443758 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.443797 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:14.443813 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.453202 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43517
I20260620 09:05:14.453223 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43517 every 8 connection(s)
I20260620 09:05:14.454198 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:14.456298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1158
I20260620 09:05:14.456388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:14.456653 392 external_mini_cluster.cc:1428] Setting key 09269bfc583f87edeaf139c70c3244e1
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:05:14.468397 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.456025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55729 (local address 127.0.98.62:42833)
0620 09:05:14.456229 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.456235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.457222 (+ 987us) server_negotiation.cc:407] Connection header received
0620 09:05:14.458611 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.458618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.458683 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.458775 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.459940 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.460490 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.461766 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.461992 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.462105 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.464567 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.464572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.464576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.464611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.466209 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.466740 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.466742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.466743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.466825 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.467115 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.467117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.467118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.467283 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.467359 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.468006 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.468148 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 09:05:14.469480 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.456556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.2:55729)
0620 09:05:14.457080 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.457135 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.458403 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.458945 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.458953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.459304 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:14.459789 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.459802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.460634 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.460637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.461605 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.461614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.462141 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.462145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.462315 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.462922 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:14.462939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:14.464396 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:14.466311 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.466316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.466328 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.466652 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.466904 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.466907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.466908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.467042 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.467396 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:14.467405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:14.467718 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.467951 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.468268 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":174,"threads_started":1}
I20260620 09:05:14.470741 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 09:05:14.470993 1287 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:14.471487 1287 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
I20260620 09:05:14.472551 962 ts_manager.cc:194] Registered new tserver with Master: 055386aa4fe74abea05107fe92ed09be (127.0.98.2:43517)
I20260620 09:05:14.473278 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:14.483709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:14.597148 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:14.597496 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:14.597584 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:14.601724 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:14.601840 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:14.601938 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:14.607627 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:14.608834 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:14.610006 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:14.620083 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:14.620085 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:14.620087 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:14.620442 1292 server_base.cc:1034] running on GCE node
I20260620 09:05:14.620901 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:14.621634 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:14.622834 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781946314622798 us; error 49 us; skew 500 ppm
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:14.625608 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:14.626533 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42325/ using document root <none> and password file <none>
I20260620 09:05:14.627130 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:14.627223 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:14.627468 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:14.629702 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "063c38ca9ced41888dd636728566850f"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "bc5badd852669c6f4e0c9c3b5d91fe99"
server_key_iv: "d2ec3deabad659902f52a7d942a2f4b4"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.630306 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "063c38ca9ced41888dd636728566850f"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "bc5badd852669c6f4e0c9c3b5d91fe99"
server_key_iv: "d2ec3deabad659902f52a7d942a2f4b4"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.634207 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:14.636835 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.638226 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:14.638381 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "063c38ca9ced41888dd636728566850f"
format_stamp: "Formatted at 2026-06-20 09:05:14 on dist-test-slave-92t4"
server_key: "bc5badd852669c6f4e0c9c3b5d91fe99"
server_key_iv: "d2ec3deabad659902f52a7d942a2f4b4"
server_key_version: "encryptionkey@0"
I20260620 09:05:14.638514 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:14.674023 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:14.676919 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:14.677152 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:14.677786 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:14.678673 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:14.678743 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.678812 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:14.678858 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:14.687683 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33971
I20260620 09:05:14.687698 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33971 every 8 connection(s)
I20260620 09:05:14.688706 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:14.690524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1292
I20260620 09:05:14.690647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:14.690948 392 external_mini_cluster.cc:1428] Setting key 967187f2784cb6456426b61177bbd4b3
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:14.702582 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.691331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43535 (local address 127.0.98.62:42833)
0620 09:05:14.691496 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.691500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.692011 (+ 511us) server_negotiation.cc:407] Connection header received
0620 09:05:14.693594 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.693597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.693648 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.693718 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.694907 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.695455 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.696597 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.696835 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.696945 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.699254 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.699258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.699261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.699298 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.700500 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.701003 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.701007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.701009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.701156 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.701493 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.701497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.701499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.701686 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.701794 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.702277 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.702408 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:05:14.703387 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.691262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42833 (local address 127.0.98.3:43535)
0620 09:05:14.691854 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.691912 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.693378 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.693895 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.693902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.694279 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:14.694763 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.694776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.695580 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.695584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.696476 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.696484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.696981 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.696985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.697195 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.697706 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:14.697721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:14.699079 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:14.700614 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.700620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.700636 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.700895 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.701249 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:14.701253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:14.701255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:14.701392 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:14.701795 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:14.701803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:14.702049 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.702239 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.702555 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260620 09:05:14.704628 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42833
I20260620 09:05:14.704893 1421 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:14.705430 1421 heartbeater.cc:511] Master 127.0.98.62:42833 requested a full tablet report, sending...
I20260620 09:05:14.706646 962 ts_manager.cc:194] Registered new tserver with Master: 063c38ca9ced41888dd636728566850f (127.0.98.3:33971)
I20260620 09:05:14.707230 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43535
I20260620 09:05:14.716804 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:05:14.728345 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.720682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.98.62:42833)
0620 09:05:14.720839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.720843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.721019 (+ 176us) server_negotiation.cc:407] Connection header received
0620 09:05:14.721213 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.721215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.721262 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.721333 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.722009 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.722534 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.723370 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.723575 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.723647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.725087 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.725092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.725095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.725125 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.726611 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.727046 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.727051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.727053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.727135 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.727416 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.727421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.727423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.727662 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.727810 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.728054 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.728173 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 09:05:14.733371 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:14.736373 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:14.751005 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.745619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.98.2:43517)
0620 09:05:14.746079 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.746089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.746275 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:05:14.746517 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.746523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.746687 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.746829 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:14.747214 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.747951 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.749754 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750314 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750443 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.750596 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.750736 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.750840 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":121,"threads_started":1}
I20260620 09:05:14.751006 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.744858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33971 (local address 127.0.0.1:59264)
0620 09:05:14.745620 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.745631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.745765 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.746537 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.746542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.746577 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:14.746742 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.746748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.748324 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.748327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.749530 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.749539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750601 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.750696 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.750719 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.750771 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.750875 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":46,"spinlock_wait_cycles":12544,"thread_start_us":102,"threads_started":1}
I20260620 09:05:14.751304 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.745169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43517 (local address 127.0.0.1:33534)
0620 09:05:14.746192 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.746204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.746340 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.747036 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.747038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.747052 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:14.747109 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.747112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.748088 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.748093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.749657 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.749663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750441 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.750556 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.750592 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.750706 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.751185 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":94,"thread_start_us":78,"threads_started":1}
I20260620 09:05:14.751543 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.744772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.0.1:60332)
0620 09:05:14.745576 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.745626 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.745765 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.746534 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.746541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.746574 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:14.746721 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.746729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.748254 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.748257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.749560 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.749569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750770 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.750862 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.750883 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.750944 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.751005 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":106,"spinlock_wait_cycles":15744,"thread_start_us":72,"threads_started":1}
I20260620 09:05:14.751544 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.745553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.98.3:33971)
0620 09:05:14.746031 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.746044 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.746069 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:05:14.746148 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.746153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.746341 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.746915 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:14.747070 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.748164 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.749656 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750463 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750873 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.751015 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.751065 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.751306 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":151,"threads_started":1}
I20260620 09:05:14.752024 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.744920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.0.98.1:46203)
0620 09:05:14.745505 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.745513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.745901 (+ 388us) server_negotiation.cc:407] Connection header received
0620 09:05:14.745985 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.745991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.746207 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.746367 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:14.746854 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.748139 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.749718 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.750463 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.750580 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.750714 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.751077 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.751878 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":79,"threads_started":1}
I20260620 09:05:14.754237 1221 tablet_service.cc:1467] Processing CreateTablet for tablet f230f06f7a4a454992582715b3121a8f (DEFAULT_TABLE table=test-table [id=450ed167e2204bd890d9f4ecd1a957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:14.754223 1355 tablet_service.cc:1467] Processing CreateTablet for tablet f230f06f7a4a454992582715b3121a8f (DEFAULT_TABLE table=test-table [id=450ed167e2204bd890d9f4ecd1a957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:14.755211 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f230f06f7a4a454992582715b3121a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:14.755264 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f230f06f7a4a454992582715b3121a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:14.756861 1087 tablet_service.cc:1467] Processing CreateTablet for tablet f230f06f7a4a454992582715b3121a8f (DEFAULT_TABLE table=test-table [id=450ed167e2204bd890d9f4ecd1a957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:14.757934 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f230f06f7a4a454992582715b3121a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:14.762068 1438 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: Bootstrap starting.
I20260620 09:05:14.763212 1439 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: Bootstrap starting.
I20260620 09:05:14.764298 1438 tablet_bootstrap.cc:654] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:14.765336 1440 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: Bootstrap starting.
I20260620 09:05:14.765367 1439 tablet_bootstrap.cc:654] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:14.765595 1438 log.cc:824] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:14.766631 1439 log.cc:824] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:14.767472 1440 tablet_bootstrap.cc:654] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:14.767925 1438 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: No bootstrap required, opened a new log
I20260620 09:05:14.768113 1438 ts_tablet_manager.cc:1397] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:05:14.768263 1440 log.cc:824] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:14.768520 1439 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: No bootstrap required, opened a new log
I20260620 09:05:14.768703 1439 ts_tablet_manager.cc:1397] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:05:14.770170 1440 tablet_bootstrap.cc:492] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: No bootstrap required, opened a new log
I20260620 09:05:14.770339 1440 ts_tablet_manager.cc:1397] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:05:14.771692 1438 raft_consensus.cc:348] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.771946 1438 raft_consensus.cc:374] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:14.771999 1438 raft_consensus.cc:729] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063c38ca9ced41888dd636728566850f, State: Initialized, Role: FOLLOWER
I20260620 09:05:14.772441 1438 consensus_queue.cc:260] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.772591 1439 raft_consensus.cc:348] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.772854 1439 raft_consensus.cc:374] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:14.772915 1439 raft_consensus.cc:729] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055386aa4fe74abea05107fe92ed09be, State: Initialized, Role: FOLLOWER
I20260620 09:05:14.773231 1421 heartbeater.cc:503] Master 127.0.98.62:42833 was elected leader, sending a full tablet report...
I20260620 09:05:14.773468 1438 ts_tablet_manager.cc:1428] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 09:05:14.773398 1439 consensus_queue.cc:260] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.774804 1287 heartbeater.cc:503] Master 127.0.98.62:42833 was elected leader, sending a full tablet report...
I20260620 09:05:14.774917 1439 ts_tablet_manager.cc:1428] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:14.774962 1440 raft_consensus.cc:348] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.775204 1440 raft_consensus.cc:374] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:14.775267 1440 raft_consensus.cc:729] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdebd03d89d64d54b4c467750db73c08, State: Initialized, Role: FOLLOWER
I20260620 09:05:14.775682 1440 consensus_queue.cc:260] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.776311 1153 heartbeater.cc:503] Master 127.0.98.62:42833 was elected leader, sending a full tablet report...
I20260620 09:05:14.776597 1440 ts_tablet_manager.cc:1428] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:14.802611 1445 raft_consensus.cc:482] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:14.802850 1445 raft_consensus.cc:504] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.804189 1445 leader_election.cc:288] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdebd03d89d64d54b4c467750db73c08 (127.0.98.1:46203), 063c38ca9ced41888dd636728566850f (127.0.98.3:33971)
I20260620 09:05:14.808562 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.804558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36617 (local address 127.0.98.3:33971)
0620 09:05:14.804699 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.804705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.805010 (+ 305us) server_negotiation.cc:407] Connection header received
0620 09:05:14.805185 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.805188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.805250 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.805326 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:14.805578 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.806291 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.807487 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.807977 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.808097 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.808161 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.808353 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.808422 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:05:14.808575 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.804404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.98.2:35497)
0620 09:05:14.804502 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.804518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.804615 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.804991 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.804994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.805014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:14.805162 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.805169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.806204 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.806208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.807372 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.807381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.808090 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.808094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.808228 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.808254 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.808347 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.808415 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":34,"spinlock_wait_cycles":7936}
I20260620 09:05:14.808568 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.804574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35497 (local address 127.0.98.1:46203)
0620 09:05:14.804700 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.804705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.804722 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:05:14.804788 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.804791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.804859 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.804947 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:14.805287 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.806097 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.807496 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.807962 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.808090 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.808159 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.808362 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.808438 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:05:14.808600 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.804404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33971 (local address 127.0.98.2:36617)
0620 09:05:14.804906 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:14.804921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:14.805071 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:14.805332 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:14.805334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:14.805355 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:14.805472 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.805478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.806417 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.806422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.807366 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:14.807375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.808096 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.808100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:14.808232 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.808250 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:05:14.808338 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:05:14.808407 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":155,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260620 09:05:14.809247 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f230f06f7a4a454992582715b3121a8f" candidate_uuid: "055386aa4fe74abea05107fe92ed09be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdebd03d89d64d54b4c467750db73c08" is_pre_election: true
I20260620 09:05:14.809262 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f230f06f7a4a454992582715b3121a8f" candidate_uuid: "055386aa4fe74abea05107fe92ed09be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063c38ca9ced41888dd636728566850f" is_pre_election: true
I20260620 09:05:14.809541 1375 raft_consensus.cc:2454] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055386aa4fe74abea05107fe92ed09be in term 0.
I20260620 09:05:14.809546 1107 raft_consensus.cc:2454] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055386aa4fe74abea05107fe92ed09be in term 0.
I20260620 09:05:14.810009 1176 leader_election.cc:302] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055386aa4fe74abea05107fe92ed09be, 063c38ca9ced41888dd636728566850f; no voters:
I20260620 09:05:14.810295 1445 raft_consensus.cc:2781] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:14.810390 1445 raft_consensus.cc:482] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:14.810449 1445 raft_consensus.cc:3037] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:14.811473 1445 raft_consensus.cc:504] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.811820 1445 leader_election.cc:288] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [CANDIDATE]: Term 1 election: Requested vote from peers cdebd03d89d64d54b4c467750db73c08 (127.0.98.1:46203), 063c38ca9ced41888dd636728566850f (127.0.98.3:33971)
I20260620 09:05:14.812184 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f230f06f7a4a454992582715b3121a8f" candidate_uuid: "055386aa4fe74abea05107fe92ed09be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063c38ca9ced41888dd636728566850f"
I20260620 09:05:14.812198 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f230f06f7a4a454992582715b3121a8f" candidate_uuid: "055386aa4fe74abea05107fe92ed09be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdebd03d89d64d54b4c467750db73c08"
I20260620 09:05:14.812320 1107 raft_consensus.cc:3037] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:14.812312 1375 raft_consensus.cc:3037] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:14.813594 1375 raft_consensus.cc:2454] T f230f06f7a4a454992582715b3121a8f P 063c38ca9ced41888dd636728566850f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055386aa4fe74abea05107fe92ed09be in term 1.
I20260620 09:05:14.813594 1107 raft_consensus.cc:2454] T f230f06f7a4a454992582715b3121a8f P cdebd03d89d64d54b4c467750db73c08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055386aa4fe74abea05107fe92ed09be in term 1.
I20260620 09:05:14.813925 1174 leader_election.cc:302] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 055386aa4fe74abea05107fe92ed09be, 063c38ca9ced41888dd636728566850f, cdebd03d89d64d54b4c467750db73c08; no voters:
I20260620 09:05:14.814123 1445 raft_consensus.cc:2781] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:14.814307 1445 raft_consensus.cc:686] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [term 1 LEADER]: Becoming Leader. State: Replica: 055386aa4fe74abea05107fe92ed09be, State: Running, Role: LEADER
I20260620 09:05:14.814623 1445 consensus_queue.cc:237] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } }
I20260620 09:05:14.817920 960 catalog_manager.cc:5495] T f230f06f7a4a454992582715b3121a8f P 055386aa4fe74abea05107fe92ed09be reported cstate change: term changed from 0 to 1, leader changed from <none> to 055386aa4fe74abea05107fe92ed09be (127.0.98.2). New cstate: current_term: 1 leader_uuid: "055386aa4fe74abea05107fe92ed09be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdebd03d89d64d54b4c467750db73c08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063c38ca9ced41888dd636728566850f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055386aa4fe74abea05107fe92ed09be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } health_report { overall_health: HEALTHY } } }
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:14.831573 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.822745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.98.1:46203)
0620 09:05:14.822876 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.822881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.823029 (+ 148us) server_negotiation.cc:407] Connection header received
0620 09:05:14.823187 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.823190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.823261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.823347 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.824044 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.824568 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.825473 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.825690 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.825805 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.828216 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.828236 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.828251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.828275 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.829825 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.830266 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.830272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.830273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.830355 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.830628 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.830633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.830634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.830898 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.831123 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.831279 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.831406 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:14.857848 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.849782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.98.1:46203)
0620 09:05:14.849967 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.849971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.850051 (+ 80us) server_negotiation.cc:407] Connection header received
0620 09:05:14.850159 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.850162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.850206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.850282 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.850928 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.851440 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.852206 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.852422 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.852495 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.854846 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.854849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.854851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.854875 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.856297 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.856647 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.856649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.856650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.856747 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.856959 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.856961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.856962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.857176 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.857274 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.857509 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.857607 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:14.882896 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:14.874760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.98.1:46203)
0620 09:05:14.874951 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:14.874955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:14.875023 (+ 68us) server_negotiation.cc:407] Connection header received
0620 09:05:14.875143 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:14.875145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:14.875192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:14.875266 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:14.875925 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.876414 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.877261 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:14.877470 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:14.877551 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:14.879873 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:14.879877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:14.879879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:14.879903 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:14.881489 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.881861 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.881864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.881865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.881936 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:14.882152 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:14.882154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:14.882155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:14.882311 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:14.882407 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:14.882572 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:05:14.882671 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 09:05:14.884370 1067 tablet_service.cc:2908] Rejecting scan request for tablet f230f06f7a4a454992582715b3121a8f: Uninitialized: clean time has not yet been initialized
I20260620 09:05:14.886502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1024
I20260620 09:05:14.892261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1158
I20260620 09:05:14.898195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1292
I20260620 09:05:14.903924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 930
2026-06-20T09:05:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2866 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[1462](info): set up 2 sockets
Jun 20 09:05:14 dist-test-slave-92t4 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 09:05:15.932811 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:16.051682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39469
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:16.166424 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:16.166708 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:16.166749 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:16.170835 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:16.170902 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:05:16.170919 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:16.170938 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:16.170955 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:16.171022 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:16.176390 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39469
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:16.177551 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:16.178645 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:16.187340 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.187330 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.187333 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.187485 1475 server_base.cc:1034] running on GCE node
I20260620 09:05:16.188271 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:16.189466 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:16.190704 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781946316190678 us; error 38 us; skew 500 ppm
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:16.193564 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:16.194478 1475 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260620 09:05:16.195047 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:16.195137 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:16.195380 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:16.197574 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "8daf0dd6e3784840938a41d8f78bc076"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "223fa437f03a1801882994ef35d91e49"
server_key_iv: "a17da0bb978bdd71dff5b26eab5ac846"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.198156 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "8daf0dd6e3784840938a41d8f78bc076"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "223fa437f03a1801882994ef35d91e49"
server_key_iv: "a17da0bb978bdd71dff5b26eab5ac846"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.201956 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:05:16.204352 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.205682 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:16.205819 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "8daf0dd6e3784840938a41d8f78bc076"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "223fa437f03a1801882994ef35d91e49"
server_key_iv: "a17da0bb978bdd71dff5b26eab5ac846"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.205933 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:16.227214 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:16.230914 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:16.231139 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:16.239164 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39469
I20260620 09:05:16.239188 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39469 every 8 connection(s)
I20260620 09:05:16.240154 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:16.243005 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:16.248142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1475
I20260620 09:05:16.248246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:16.248512 392 external_mini_cluster.cc:1428] Setting key 08158e1dda10322ba203bec51ff33463
I20260620 09:05:16.249361 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Bootstrap starting.
I20260620 09:05:16.251777 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:16.252544 1553 log.cc:824] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:16.254544 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: No bootstrap required, opened a new log
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:16.258424 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } }
I20260620 09:05:16.258657 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:16.258726 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8daf0dd6e3784840938a41d8f78bc076, State: Initialized, Role: FOLLOWER
I20260620 09:05:16.259339 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } }
I20260620 09:05:16.259527 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:16.259601 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:16.259707 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:16.260915 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } }
I20260620 09:05:16.261324 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8daf0dd6e3784840938a41d8f78bc076; no voters:
I20260620 09:05:16.261677 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:16.261934 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:16.262187 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [term 1 LEADER]: Becoming Leader. State: Replica: 8daf0dd6e3784840938a41d8f78bc076, State: Running, Role: LEADER
I20260620 09:05:16.262650 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } }
I20260620 09:05:16.263038 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:16.263782 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8daf0dd6e3784840938a41d8f78bc076. Latest consensus state: current_term: 1 leader_uuid: "8daf0dd6e3784840938a41d8f78bc076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } } }
I20260620 09:05:16.263897 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:16.264096 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.250047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.98.62:39469)
0620 09:05:16.250482 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.250497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.250532 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:16.251190 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.251207 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.251551 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.251904 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.252672 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.253910 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.254811 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.255282 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.255450 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.258216 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.258221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.258235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.258263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.260347 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.260808 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.260815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.260821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.260967 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.261290 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.261293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.261295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.261847 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.262610 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.262759 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.263071 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
I20260620 09:05:16.264317 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:16.264575 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8daf0dd6e3784840938a41d8f78bc076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf0dd6e3784840938a41d8f78bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39469 } } }
I20260620 09:05:16.264731 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:16.267656 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:16.273241 1567 catalog_manager.cc:1347] Generated new cluster ID: 77bd895f337347a181efe35174205c7b
I20260620 09:05:16.273321 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:16.281817 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:16.282779 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:16.292546 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Generated new TSK 0
I20260620 09:05:16.293298 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:16.320412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39469
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:16.434744 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:16.435027 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:16.435074 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:16.439309 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:16.439373 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:16.439414 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:16.439448 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:16.444734 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:16.445921 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:16.446919 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:16.456625 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.456609 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.456939 1579 server_base.cc:1034] running on GCE node
W20260620 09:05:16.457060 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.457432 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:16.458094 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:16.459295 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781946316459283 us; error 31 us; skew 500 ppm
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:16.462222 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:16.463267 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46123/ using document root <none> and password file <none>
I20260620 09:05:16.463891 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:16.463982 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:16.464179 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:16.466405 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "de6682c6a7fe4ac1b34dbd7576fd6877"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "3075c261442f7d09e6ee8673e80c2eb4"
server_key_iv: "dcd75cc56a5d01e8acc54a172f955ada"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.466925 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "de6682c6a7fe4ac1b34dbd7576fd6877"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "3075c261442f7d09e6ee8673e80c2eb4"
server_key_iv: "dcd75cc56a5d01e8acc54a172f955ada"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.470714 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:05:16.473196 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.474432 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:16.474560 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "de6682c6a7fe4ac1b34dbd7576fd6877"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "3075c261442f7d09e6ee8673e80c2eb4"
server_key_iv: "dcd75cc56a5d01e8acc54a172f955ada"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.474668 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:16.493211 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:16.496757 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:16.496955 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:16.498570 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:16.498620 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.498663 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:16.498685 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:16.515224 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41697
I20260620 09:05:16.515305 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41697 every 8 connection(s)
I20260620 09:05:16.516441 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:16.517681 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.499491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44227 (local address 127.0.98.62:39469)
0620 09:05:16.499691 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.499698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.500646 (+ 948us) server_negotiation.cc:407] Connection header received
0620 09:05:16.502172 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.502176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.502239 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.502310 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.504477 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.505517 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.508090 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.508479 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.508582 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.513357 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.513364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.513370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.513408 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.515244 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.515859 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.515862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.515864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.515947 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.516334 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.516337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.516338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.516570 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.516659 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.517389 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.517518 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:05:16.518399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1579
I20260620 09:05:16.518528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:16.518628 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.499916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.1:44227)
0620 09:05:16.500435 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.500490 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.501767 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.502815 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.502828 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.503415 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.504260 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.504288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.506361 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.506366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.507921 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.507933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.508899 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.508903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.509264 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.510965 (+ 1701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.510992 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.513099 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.515394 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.515400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.515416 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.515739 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.516048 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.516054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.516058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.516233 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.516665 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.516672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.517017 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.517343 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.517677 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":157,"threads_started":1}
I20260620 09:05:16.518862 392 external_mini_cluster.cc:1428] Setting key 1a5fe84b6e055723ccc4ac59c226049e
I20260620 09:05:16.520295 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39469
I20260620 09:05:16.520459 1710 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:16.521066 1710 heartbeater.cc:511] Master 127.0.98.62:39469 requested a full tablet report, sending...
I20260620 09:05:16.523360 1505 ts_manager.cc:194] Registered new tserver with Master: de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1:41697)
I20260620 09:05:16.524791 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44227
I20260620 09:05:16.530666 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.522396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38325 (local address 127.0.98.62:39469)
0620 09:05:16.522547 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.522550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.523004 (+ 454us) server_negotiation.cc:407] Connection header received
0620 09:05:16.523128 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.523131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.523169 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.523241 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.523809 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.524337 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.525622 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.526076 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.526211 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.527500 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.527506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.527512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.527536 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.528967 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.529478 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.529481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.529482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.529561 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.529887 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.529890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.529891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.530054 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.530127 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.530450 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.530545 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:05:16.530886 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.522325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.1:38325)
0620 09:05:16.522897 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.522915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.523035 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.523300 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.523304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.523557 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.523692 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.523699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.524471 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.524474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.525460 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.525474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.526235 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.526239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.526407 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.526901 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.526915 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.527337 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.529109 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.529115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.529119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.529381 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.529658 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.529661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.529664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.529786 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.530149 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.530154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.530279 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.530411 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.530590 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":285,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:16.550223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39469
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:16.663192 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:16.663570 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:16.663666 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:16.667917 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:16.667981 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:16.668058 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:16.668126 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:16.673491 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:16.674615 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:16.675710 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:16.685928 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.685916 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.685938 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.686242 1719 server_base.cc:1034] running on GCE node
I20260620 09:05:16.686748 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:16.687369 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:16.688524 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781946316688494 us; error 45 us; skew 500 ppm
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:16.691315 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:16.692281 1719 webserver.cc:460] Webserver started at http://127.0.98.2:44445/ using document root <none> and password file <none>
I20260620 09:05:16.692866 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:16.693225 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:16.693483 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:16.695647 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "9fb594f1faef4721b1b8f3e4941297aa"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "4f24c4bbb7c7950fbc9218837c856844"
server_key_iv: "6dca09d4675bcdba80f9b5240b497fed"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.696295 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb594f1faef4721b1b8f3e4941297aa"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "4f24c4bbb7c7950fbc9218837c856844"
server_key_iv: "6dca09d4675bcdba80f9b5240b497fed"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.700270 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 09:05:16.702852 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.704089 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:16.704242 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "9fb594f1faef4721b1b8f3e4941297aa"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "4f24c4bbb7c7950fbc9218837c856844"
server_key_iv: "6dca09d4675bcdba80f9b5240b497fed"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.704365 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:16.738224 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:16.741333 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:16.741544 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:16.743083 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:16.743157 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.743242 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:16.743286 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:16.754570 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41953
I20260620 09:05:16.754932 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41953 every 8 connection(s)
I20260620 09:05:16.755579 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:16.757251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1719
I20260620 09:05:16.757391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:16.757690 392 external_mini_cluster.cc:1428] Setting key 650eee919dedbf2596b832a956af426e
I20260620 09:05:16.758229 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.743872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45369 (local address 127.0.98.62:39469)
0620 09:05:16.744018 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.744021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.745117 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 09:05:16.746176 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.746179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.746243 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.746323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.748172 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.749101 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.750517 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.750933 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.751056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.754204 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.754209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.754212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.754243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.756004 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.756626 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.756629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.756630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.756711 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.756997 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.756999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.757000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.757236 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.757317 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.757942 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.758049 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:05:16.759461 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.744448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.2:45369)
0620 09:05:16.744942 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.744986 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.745933 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.746786 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.746797 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.747272 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.747987 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.748008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.749239 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.749243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.750380 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.750390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.751120 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.751125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.751312 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.751980 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.751996 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.753996 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.756134 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.756140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.756155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.756520 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.756795 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.756797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.756799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.756924 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.757356 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.757365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.757694 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.757917 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.758306 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":161,"threads_started":1}
I20260620 09:05:16.760811 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39469
I20260620 09:05:16.760967 1850 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:16.761615 1850 heartbeater.cc:511] Master 127.0.98.62:39469 requested a full tablet report, sending...
I20260620 09:05:16.763137 1505 ts_manager.cc:194] Registered new tserver with Master: 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953)
I20260620 09:05:16.763840 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45369
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:16.771368 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.764152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37243 (local address 127.0.98.62:39469)
0620 09:05:16.764252 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.764256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.764268 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:16.764320 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.764322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.764363 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.764439 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.764989 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.765591 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.766703 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.766908 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.766983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.768385 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.768388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.768390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.768410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.769934 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.770357 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.770359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.770361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.770427 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.770649 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.770651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.770653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.770886 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.770991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.771164 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.771250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 09:05:16.771375 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.762914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.2:37243)
0620 09:05:16.763201 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.763250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.763334 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.764467 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.764473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.764726 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.764853 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.764860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.765723 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.765727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.766584 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.766596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.767022 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.767028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.767162 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.767745 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.767763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.768256 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.770037 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.770044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.770048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.770275 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.770503 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.770505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.770507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.770578 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.770969 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.770971 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.771047 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.771127 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.771236 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:16.787573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39469
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:16.901300 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:16.901569 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:16.901616 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:16.905784 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:16.905848 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:16.905889 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:16.905931 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:16.911413 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:16.912459 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:16.913568 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:16.923403 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:16.923403 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.923476 1859 server_base.cc:1034] running on GCE node
W20260620 09:05:16.923404 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:16.924073 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:16.924762 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:16.925952 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781946316925919 us; error 55 us; skew 500 ppm
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:16.928714 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:16.929687 1859 webserver.cc:460] Webserver started at http://127.0.98.3:39197/ using document root <none> and password file <none>
I20260620 09:05:16.930243 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:16.930303 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:16.930472 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:16.932565 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "834bdc5131984d17979d9d266cd56f72"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "db4606e74f373255ead2a6e6f6e61893"
server_key_iv: "40fe46835b56401fb1668098fb5f27fa"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.933148 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "834bdc5131984d17979d9d266cd56f72"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "db4606e74f373255ead2a6e6f6e61893"
server_key_iv: "40fe46835b56401fb1668098fb5f27fa"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.936961 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 09:05:16.939467 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.940658 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:16.940783 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "834bdc5131984d17979d9d266cd56f72"
format_stamp: "Formatted at 2026-06-20 09:05:16 on dist-test-slave-92t4"
server_key: "db4606e74f373255ead2a6e6f6e61893"
server_key_iv: "40fe46835b56401fb1668098fb5f27fa"
server_key_version: "encryptionkey@0"
I20260620 09:05:16.940893 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:16.963785 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:16.966912 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:16.967128 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:16.968662 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:16.968737 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:16.968811 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:16.968860 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:16 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:16.979883 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33639
I20260620 09:05:16.979924 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33639 every 8 connection(s)
I20260620 09:05:16.981079 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:16.982880 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.969440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40689 (local address 127.0.98.62:39469)
0620 09:05:16.969579 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.969585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.970688 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 09:05:16.971626 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.971630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.971687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.971776 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.973260 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.973881 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.975183 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.975483 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.975597 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.979051 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.979055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.979058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.979087 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.980819 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.981406 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.981413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.981418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.981521 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.981927 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.981931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.981933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.982120 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.982197 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.982652 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.982750 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:05:16.983719 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.970042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.3:40689)
0620 09:05:16.970537 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.970581 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.971422 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.971946 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.971954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.972320 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.973119 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.973140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.974031 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.974035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.975059 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.975068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.975645 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.975649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.975854 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.976542 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.976560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.978873 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.980927 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.980932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.980943 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.981301 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.981637 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.981641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.981643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.981820 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.982221 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.982226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.982448 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.982593 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.982842 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":140,"threads_started":1}
I20260620 09:05:16.984370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 1859
I20260620 09:05:16.984503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:16.984804 392 external_mini_cluster.cc:1428] Setting key f16c2ccd651d187fc0f88cccdccc32b9
I20260620 09:05:16.987267 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39469
I20260620 09:05:16.987407 1990 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:16.987879 1990 heartbeater.cc:511] Master 127.0.98.62:39469 requested a full tablet report, sending...
I20260620 09:05:16.988842 1506 ts_manager.cc:194] Registered new tserver with Master: 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639)
I20260620 09:05:16.989596 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40689
I20260620 09:05:16.994874 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.987463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44867 (local address 127.0.98.62:39469)
0620 09:05:16.987732 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:16.987736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:16.988210 (+ 474us) server_negotiation.cc:407] Connection header received
0620 09:05:16.988287 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:16.988290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:16.988336 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:16.988422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:16.988861 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.989619 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.990590 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.990898 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.991009 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.992101 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:16.992104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:16.992106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:16.992126 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:16.993557 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.993953 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.993956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.993957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.994023 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:16.994249 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:16.994252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:16.994253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:16.994405 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:16.994477 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:16.994669 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:05:16.994753 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260620 09:05:16.994882 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:16.987397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.98.3:44867)
0620 09:05:16.988126 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:16.988139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:16.988222 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:16.988441 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:16.988444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:16.988659 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:16.988755 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.988760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.989780 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.989786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.990464 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:16.990473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:16.991006 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:16.991008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:16.991073 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:16.991596 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:16.991607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:16.991984 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:16.993670 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.993673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.993674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.993871 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.994097 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:16.994099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:16.994100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:16.994178 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:16.994481 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:16.994483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:16.994552 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:05:16.994636 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:05:16.994738 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260620 09:05:16.998584 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:17.034646 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.025746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.0.98.62:39469)
0620 09:05:17.025900 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.025904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.025919 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:05:17.025970 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.025973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.026020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.026146 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:17.027115 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.027680 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.028592 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.028822 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.028943 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.031372 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:17.031375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:17.031377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:17.031405 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:17.032990 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.033458 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.033461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.033462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.033536 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.033788 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.033790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.033791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.033959 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:17.034101 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.034300 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.034456 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:05:17.037676 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:17.040681 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:17.055116 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.050198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.98.2:41953)
0620 09:05:17.050509 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.050520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.050935 (+ 415us) server_negotiation.cc:407] Connection header received
0620 09:05:17.051008 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.051012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.051179 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.051324 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.051542 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.052429 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.053868 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.054434 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.054557 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.054730 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.054864 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.054956 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260620 09:05:17.055106 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.050021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41953 (local address 127.0.0.1:53528)
0620 09:05:17.050732 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.050746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.050842 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.051321 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.051324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.051341 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.051428 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.051433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.052536 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.052541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.053731 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.053744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.054554 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.054558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.054671 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.054709 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.054834 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.054950 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":76,"thread_start_us":87,"threads_started":1}
I20260620 09:05:17.055562 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.049097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.0.98.3:33639)
0620 09:05:17.049539 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.049551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.049748 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:05:17.049981 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.049988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.050202 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.050340 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.050832 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.051979 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.053899 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.054696 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.054777 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.054921 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.055335 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.055426 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":113,"threads_started":1}
I20260620 09:05:17.055932 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.048925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33639 (local address 127.0.0.1:60164)
0620 09:05:17.049568 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.049659 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.049802 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.050531 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.050536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.050557 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.050732 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.050740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.052077 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.052080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.053746 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.053756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.055004 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.055008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.055160 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.055188 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.055725 (+ 537us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.055806 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":217,"thread_start_us":203,"threads_started":1}
I20260620 09:05:17.056031 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.049623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.0.98.1:41697)
0620 09:05:17.050036 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.050044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.050066 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:05:17.050148 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.050153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.050352 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.050495 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.051061 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.051910 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.053915 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.054695 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.054777 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.054906 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.055796 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.055893 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":83,"threads_started":1}
I20260620 09:05:17.056392 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.049216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.0.1:33172)
0620 09:05:17.049668 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.049682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.049802 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.050668 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.050671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.050683 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.050943 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.050948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.052018 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.052023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.053780 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.053789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.055004 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.055008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.055552 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.055590 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.056201 (+ 611us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.056275 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":644,"thread_start_us":88,"threads_started":1}
I20260620 09:05:17.058485 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbadc2a44fd44938db899a2c668a71f (DEFAULT_TABLE table=test-table [id=1ebc48094935481887c339a1bd0b679b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.059494 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbadc2a44fd44938db899a2c668a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.059089 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbadc2a44fd44938db899a2c668a71f (DEFAULT_TABLE table=test-table [id=1ebc48094935481887c339a1bd0b679b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.059075 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbadc2a44fd44938db899a2c668a71f (DEFAULT_TABLE table=test-table [id=1ebc48094935481887c339a1bd0b679b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.060109 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbadc2a44fd44938db899a2c668a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.060109 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbadc2a44fd44938db899a2c668a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.066897 2013 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Bootstrap starting.
I20260620 09:05:17.067963 2014 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Bootstrap starting.
I20260620 09:05:17.069329 2013 tablet_bootstrap.cc:654] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.069870 2014 tablet_bootstrap.cc:654] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.070192 2013 log.cc:824] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:17.070567 2014 log.cc:824] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:17.071197 2015 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Bootstrap starting.
I20260620 09:05:17.072168 2014 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: No bootstrap required, opened a new log
I20260620 09:05:17.072288 2013 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: No bootstrap required, opened a new log
I20260620 09:05:17.072386 2014 ts_tablet_manager.cc:1397] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:05:17.072461 2013 ts_tablet_manager.cc:1397] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:05:17.073243 2015 tablet_bootstrap.cc:654] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.074025 2015 log.cc:824] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:17.076103 2015 tablet_bootstrap.cc:492] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: No bootstrap required, opened a new log
I20260620 09:05:17.076304 2015 ts_tablet_manager.cc:1397] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:05:17.076289 2013 raft_consensus.cc:348] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.076583 2013 raft_consensus.cc:374] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.076637 2013 raft_consensus.cc:729] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb594f1faef4721b1b8f3e4941297aa, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.076613 2014 raft_consensus.cc:348] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.076879 2014 raft_consensus.cc:374] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.076933 2014 raft_consensus.cc:729] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6682c6a7fe4ac1b34dbd7576fd6877, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.077160 2013 consensus_queue.cc:260] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.077366 2014 consensus_queue.cc:260] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.077860 1850 heartbeater.cc:503] Master 127.0.98.62:39469 was elected leader, sending a full tablet report...
I20260620 09:05:17.078246 2013 ts_tablet_manager.cc:1428] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:17.078254 2014 ts_tablet_manager.cc:1428] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:05:17.078578 1710 heartbeater.cc:503] Master 127.0.98.62:39469 was elected leader, sending a full tablet report...
I20260620 09:05:17.080298 2015 raft_consensus.cc:348] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.080559 2015 raft_consensus.cc:374] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.080618 2015 raft_consensus.cc:729] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834bdc5131984d17979d9d266cd56f72, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.081075 2015 consensus_queue.cc:260] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.082127 1990 heartbeater.cc:503] Master 127.0.98.62:39469 was elected leader, sending a full tablet report...
I20260620 09:05:17.083262 2015 ts_tablet_manager.cc:1428] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 09:05:17.168576 2019 raft_consensus.cc:482] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:17.168843 2019 raft_consensus.cc:504] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.170149 2019 leader_election.cc:288] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639), de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1:41697)
I20260620 09:05:17.174990 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.170512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55665 (local address 127.0.98.3:33639)
0620 09:05:17.170650 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.170656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.170762 (+ 106us) server_negotiation.cc:407] Connection header received
0620 09:05:17.170884 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.170887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.170962 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.171048 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.171419 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.172210 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.173953 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.174424 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.174545 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.174612 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.174796 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.174864 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:05:17.175010 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.170387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33639 (local address 127.0.98.2:55665)
0620 09:05:17.170647 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.170664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.170770 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.171071 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.171073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.171086 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.171264 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.171273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.172339 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.172343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.173759 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.173770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.174549 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.174552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.174663 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.174686 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.174783 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.174865 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":36}
I20260620 09:05:17.175187 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.170498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.98.2:48049)
0620 09:05:17.170853 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.170866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.170950 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.171161 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.171163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.171174 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.171293 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.171301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.172339 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.172343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.174043 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.174049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.174746 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.174750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.174851 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.174876 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.174966 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.175033 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":22,"thread_start_us":111,"threads_started":1}
I20260620 09:05:17.175175 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.170579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48049 (local address 127.0.98.1:41697)
0620 09:05:17.170707 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.170712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.170957 (+ 245us) server_negotiation.cc:407] Connection header received
0620 09:05:17.171011 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.171015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.171067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.171133 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.171419 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.172210 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.174185 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.174613 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.174738 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.174795 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.174982 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.175059 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:05:17.175799 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbadc2a44fd44938db899a2c668a71f" candidate_uuid: "9fb594f1faef4721b1b8f3e4941297aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72" is_pre_election: true
I20260620 09:05:17.175797 1652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbadc2a44fd44938db899a2c668a71f" candidate_uuid: "9fb594f1faef4721b1b8f3e4941297aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" is_pre_election: true
I20260620 09:05:17.176071 1944 raft_consensus.cc:2454] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb594f1faef4721b1b8f3e4941297aa in term 0.
I20260620 09:05:17.176081 1652 raft_consensus.cc:2454] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb594f1faef4721b1b8f3e4941297aa in term 0.
I20260620 09:05:17.176517 1735 leader_election.cc:302] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fb594f1faef4721b1b8f3e4941297aa, de6682c6a7fe4ac1b34dbd7576fd6877; no voters:
I20260620 09:05:17.176776 2019 raft_consensus.cc:2781] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:17.176855 2019 raft_consensus.cc:482] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:17.176898 2019 raft_consensus.cc:3037] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.177930 2019 raft_consensus.cc:504] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.178293 2019 leader_election.cc:288] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [CANDIDATE]: Term 1 election: Requested vote from peers 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639), de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1:41697)
I20260620 09:05:17.178606 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbadc2a44fd44938db899a2c668a71f" candidate_uuid: "9fb594f1faef4721b1b8f3e4941297aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72"
I20260620 09:05:17.178624 1652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbadc2a44fd44938db899a2c668a71f" candidate_uuid: "9fb594f1faef4721b1b8f3e4941297aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877"
I20260620 09:05:17.178737 1652 raft_consensus.cc:3037] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.178738 1944 raft_consensus.cc:3037] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.180078 1944 raft_consensus.cc:2454] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb594f1faef4721b1b8f3e4941297aa in term 1.
I20260620 09:05:17.180078 1652 raft_consensus.cc:2454] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb594f1faef4721b1b8f3e4941297aa in term 1.
I20260620 09:05:17.180430 1736 leader_election.cc:302] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 834bdc5131984d17979d9d266cd56f72, 9fb594f1faef4721b1b8f3e4941297aa; no voters:
I20260620 09:05:17.180662 2019 raft_consensus.cc:2781] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:17.180909 2019 raft_consensus.cc:686] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 1 LEADER]: Becoming Leader. State: Replica: 9fb594f1faef4721b1b8f3e4941297aa, State: Running, Role: LEADER
I20260620 09:05:17.181352 2019 consensus_queue.cc:237] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260620 09:05:17.184541 1507 catalog_manager.cc:5495] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9fb594f1faef4721b1b8f3e4941297aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: UNKNOWN } } }
I20260620 09:05:17.216487 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.213119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.0.98.2:41953)
0620 09:05:17.213239 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.213242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.213273 (+ 31us) server_negotiation.cc:407] Connection header received
0620 09:05:17.213384 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.213386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.213440 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.213523 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:17.213763 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.214330 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.215167 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.215383 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.215457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.215741 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.216159 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.216241 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.216311 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.216376 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:05:17.222528 1652 raft_consensus.cc:1261] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Refusing update from remote peer 9fb594f1faef4721b1b8f3e4941297aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.222855 1944 raft_consensus.cc:1261] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Refusing update from remote peer 9fb594f1faef4721b1b8f3e4941297aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.223301 2023 consensus_queue.cc:1035] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.223477 2019 consensus_queue.cc:1035] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.230976 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298852115335405568 to 7298852115177349120. Current Snapshot: MvccSnapshot[applied={T|T < 7298852115335405568}]
I20260620 09:05:17.242449 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:17.242619 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:17.245283 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Sending DeleteTablet for 3 replicas of tablet 8cbadc2a44fd44938db899a2c668a71f
I20260620 09:05:17.246127 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbadc2a44fd44938db899a2c668a71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53528
I20260620 09:05:17.246171 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbadc2a44fd44938db899a2c668a71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33172
I20260620 09:05:17.246418 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbadc2a44fd44938db899a2c668a71f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60164
I20260620 09:05:17.247174 2037 tablet_replica.cc:330] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: stopping tablet replica
I20260620 09:05:17.247370 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:05:17.247491 2037 raft_consensus.cc:2229] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 09:05:17.247789 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:17.247848 2037 raft_consensus.cc:2258] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.248335 2036 tablet_replica.cc:330] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: stopping tablet replica
I20260620 09:05:17.249285 2038 tablet_replica.cc:330] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: stopping tablet replica
I20260620 09:05:17.249665 2038 raft_consensus.cc:2229] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:17.251078 2037 ts_tablet_manager.cc:1905] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.253928 2037 ts_tablet_manager.cc:1918] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:17.254019 2037 log.cc:1196] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/8cbadc2a44fd44938db899a2c668a71f
I20260620 09:05:17.253993 1642 tablet_service.cc:1467] Processing CreateTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 (DEFAULT_TABLE table=test-table [id=6724c8b5e9644fc8897534c078ec9cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.254248 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122921b6fc54bd897603cd3f6bc25b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.254350 2037 ts_tablet_manager.cc:1939] T 8cbadc2a44fd44938db899a2c668a71f P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting consensus metadata
I20260620 09:05:17.255664 2036 raft_consensus.cc:2229] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:17.256098 2036 raft_consensus.cc:2258] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.256367 2014 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Bootstrap starting.
I20260620 09:05:17.256882 2038 raft_consensus.cc:2258] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.256937 1784 tablet_service.cc:1467] Processing CreateTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 (DEFAULT_TABLE table=test-table [id=6724c8b5e9644fc8897534c078ec9cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.257242 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122921b6fc54bd897603cd3f6bc25b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.257453 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 (DEFAULT_TABLE table=test-table [id=6724c8b5e9644fc8897534c078ec9cf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:17.257645 1491 catalog_manager.cc:4841] TS de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1:41697): tablet 8cbadc2a44fd44938db899a2c668a71f (table test-table [id=1ebc48094935481887c339a1bd0b679b]) successfully deleted
I20260620 09:05:17.257699 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b122921b6fc54bd897603cd3f6bc25b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.258915 2014 tablet_bootstrap.cc:654] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.259727 2013 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Bootstrap starting.
I20260620 09:05:17.260062 2015 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Bootstrap starting.
I20260620 09:05:17.260735 2013 tablet_bootstrap.cc:654] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.260932 2038 ts_tablet_manager.cc:1905] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.261021 2015 tablet_bootstrap.cc:654] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.261962 2013 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: No bootstrap required, opened a new log
I20260620 09:05:17.262061 2013 ts_tablet_manager.cc:1397] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 09:05:17.262428 2013 raft_consensus.cc:348] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.262538 2013 raft_consensus.cc:374] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.262578 2013 raft_consensus.cc:729] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb594f1faef4721b1b8f3e4941297aa, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.262688 2013 consensus_queue.cc:260] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.263026 2013 ts_tablet_manager.cc:1428] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:17.263392 2038 ts_tablet_manager.cc:1918] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:17.263479 2038 log.cc:1196] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/8cbadc2a44fd44938db899a2c668a71f
I20260620 09:05:17.263779 2038 ts_tablet_manager.cc:1939] T 8cbadc2a44fd44938db899a2c668a71f P 834bdc5131984d17979d9d266cd56f72: Deleting consensus metadata
I20260620 09:05:17.264667 2036 ts_tablet_manager.cc:1905] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.265110 1492 catalog_manager.cc:4841] TS 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639): tablet 8cbadc2a44fd44938db899a2c668a71f (table test-table [id=1ebc48094935481887c339a1bd0b679b]) successfully deleted
I20260620 09:05:17.265743 2015 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: No bootstrap required, opened a new log
I20260620 09:05:17.265830 2015 ts_tablet_manager.cc:1397] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 09:05:17.266232 2015 raft_consensus.cc:348] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.266331 2015 raft_consensus.cc:374] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.266360 2015 raft_consensus.cc:729] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834bdc5131984d17979d9d266cd56f72, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.266484 2015 consensus_queue.cc:260] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.266743 2015 ts_tablet_manager.cc:1428] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:17.267690 2014 tablet_bootstrap.cc:492] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: No bootstrap required, opened a new log
I20260620 09:05:17.267701 2036 ts_tablet_manager.cc:1918] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:17.267793 2014 ts_tablet_manager.cc:1397] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 09:05:17.267807 2036 log.cc:1196] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/8cbadc2a44fd44938db899a2c668a71f
I20260620 09:05:17.268121 2036 ts_tablet_manager.cc:1939] T 8cbadc2a44fd44938db899a2c668a71f P 9fb594f1faef4721b1b8f3e4941297aa: Deleting consensus metadata
I20260620 09:05:17.268323 2014 raft_consensus.cc:348] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.268425 2014 raft_consensus.cc:374] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.268455 2014 raft_consensus.cc:729] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6682c6a7fe4ac1b34dbd7576fd6877, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.268581 2014 consensus_queue.cc:260] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.268816 2014 ts_tablet_manager.cc:1428] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:17.269762 1492 catalog_manager.cc:4841] TS 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953): tablet 8cbadc2a44fd44938db899a2c668a71f (table test-table [id=1ebc48094935481887c339a1bd0b679b]) successfully deleted
W20260620 09:05:17.482821 1991 tablet.cc:2369] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:05:17.506963 1851 tablet.cc:2369] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:05:17.518070 1711 tablet.cc:2369] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:17.575919 2020 raft_consensus.cc:482] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:17.576092 2020 raft_consensus.cc:504] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.577389 2020 leader_election.cc:288] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953), 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639)
I20260620 09:05:17.582337 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.577804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56125 (local address 127.0.98.2:41953)
0620 09:05:17.577947 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.577952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.578119 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:05:17.578263 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.578267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.578321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.578401 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.578690 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.579473 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581071 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.581684 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581790 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.581858 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.582136 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.582206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:05:17.582337 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.578218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39849 (local address 127.0.98.3:33639)
0620 09:05:17.578339 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.578345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.578512 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:05:17.578576 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.578579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.578618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.578715 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:17.578975 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.579694 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581070 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.581678 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581790 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.581855 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.582145 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.582214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:05:17.582410 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.577660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41953 (local address 127.0.98.1:56125)
0620 09:05:17.577996 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.578010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.578156 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.578421 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.578426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.578451 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.578575 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.578582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.579582 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.579584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.580900 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.580911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.581805 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.581989 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.582030 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.582105 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.582217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":72,"thread_start_us":118,"threads_started":1}
I20260620 09:05:17.582602 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.578115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33639 (local address 127.0.98.1:39849)
0620 09:05:17.578410 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.578421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.578505 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.578741 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.578743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.578759 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:17.578840 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.578844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.579809 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.579812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.580920 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.580925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.581805 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.581809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.581992 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.582035 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.582105 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.582212 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":104,"thread_start_us":85,"threads_started":1}
I20260620 09:05:17.582921 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122921b6fc54bd897603cd3f6bc25b6" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb594f1faef4721b1b8f3e4941297aa" is_pre_election: true
I20260620 09:05:17.583091 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122921b6fc54bd897603cd3f6bc25b6" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72" is_pre_election: true
I20260620 09:05:17.583181 1804 raft_consensus.cc:2454] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 0.
I20260620 09:05:17.583245 1944 raft_consensus.cc:2454] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 0.
I20260620 09:05:17.583596 1596 leader_election.cc:302] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 834bdc5131984d17979d9d266cd56f72, de6682c6a7fe4ac1b34dbd7576fd6877; no voters:
I20260620 09:05:17.583870 2020 raft_consensus.cc:2781] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:17.583925 2020 raft_consensus.cc:482] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:17.583952 2020 raft_consensus.cc:3037] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.585556 2020 raft_consensus.cc:504] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.585901 2020 leader_election.cc:288] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 election: Requested vote from peers 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953), 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639)
I20260620 09:05:17.586244 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122921b6fc54bd897603cd3f6bc25b6" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb594f1faef4721b1b8f3e4941297aa"
I20260620 09:05:17.586313 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b122921b6fc54bd897603cd3f6bc25b6" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72"
I20260620 09:05:17.586388 1804 raft_consensus.cc:3037] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.586438 1944 raft_consensus.cc:3037] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.587323 1944 raft_consensus.cc:2454] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 1.
I20260620 09:05:17.587335 1804 raft_consensus.cc:2454] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 1.
I20260620 09:05:17.587591 1596 leader_election.cc:302] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 834bdc5131984d17979d9d266cd56f72, de6682c6a7fe4ac1b34dbd7576fd6877; no voters:
I20260620 09:05:17.587796 2020 raft_consensus.cc:2781] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:17.587996 2020 raft_consensus.cc:686] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 LEADER]: Becoming Leader. State: Replica: de6682c6a7fe4ac1b34dbd7576fd6877, State: Running, Role: LEADER
I20260620 09:05:17.588266 2020 consensus_queue.cc:237] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.590670 1507 catalog_manager.cc:5495] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 reported cstate change: term changed from 0 to 1, leader changed from <none> to de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:17.647089 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.638831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39469 (local address 127.0.0.1:39254)
0620 09:05:17.639136 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.639159 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.639259 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.639502 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.639504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.639903 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:17.639998 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.640004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.640726 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.640728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.641291 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.641299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.641705 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.641707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.641765 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.642285 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:17.642297 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:17.643704 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:17.645511 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:17.645520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:17.645527 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:17.645745 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:17.646026 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:17.646029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:17.646031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:17.646107 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:17.646492 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:17.646501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:17.646610 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.646722 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.646876 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":103,"threads_started":1}
I20260620 09:05:17.647096 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.638933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.0.98.62:39469)
0620 09:05:17.639215 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.639221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.639236 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:05:17.639344 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.639346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.639386 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.639465 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:17.640094 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.640613 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.641384 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.641594 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.641669 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.643837 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:17.643847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:17.643850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:17.643868 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:17.645385 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.645851 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.645855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.645858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.645931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.646198 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.646201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.646202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.646379 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:17.646480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.646783 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.646931 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260620 09:05:17.651000 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39254:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:05:17.651746 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:17.656286 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 1823ae494f954cb89bc30ec5aab16996 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=833513de12164ca0ab49231e009e1db4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:17.656319 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 1823ae494f954cb89bc30ec5aab16996 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=833513de12164ca0ab49231e009e1db4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:17.656319 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1823ae494f954cb89bc30ec5aab16996 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=833513de12164ca0ab49231e009e1db4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:17.656701 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1823ae494f954cb89bc30ec5aab16996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.656710 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1823ae494f954cb89bc30ec5aab16996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.656713 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1823ae494f954cb89bc30ec5aab16996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:17.659246 2014 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: Bootstrap starting.
I20260620 09:05:17.659468 2015 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: Bootstrap starting.
I20260620 09:05:17.660013 2013 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: Bootstrap starting.
I20260620 09:05:17.660274 2014 tablet_bootstrap.cc:654] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.660365 2015 tablet_bootstrap.cc:654] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.660887 2013 tablet_bootstrap.cc:654] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:17.661649 2014 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: No bootstrap required, opened a new log
I20260620 09:05:17.661756 2014 ts_tablet_manager.cc:1397] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:05:17.661916 2015 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: No bootstrap required, opened a new log
I20260620 09:05:17.662003 2015 ts_tablet_manager.cc:1397] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:05:17.662237 2014 raft_consensus.cc:348] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.662379 2014 raft_consensus.cc:374] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.662423 2014 raft_consensus.cc:729] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6682c6a7fe4ac1b34dbd7576fd6877, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.662451 2015 raft_consensus.cc:348] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.662599 2015 raft_consensus.cc:374] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.662590 2014 consensus_queue.cc:260] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.662658 2015 raft_consensus.cc:729] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834bdc5131984d17979d9d266cd56f72, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.662791 2015 consensus_queue.cc:260] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.662844 2020 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.662905 2014 ts_tablet_manager.cc:1428] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:17.662976 2020 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.663064 2015 ts_tablet_manager.cc:1428] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:17.663010 2021 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.663134 2021 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.663811 2013 tablet_bootstrap.cc:492] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: No bootstrap required, opened a new log
I20260620 09:05:17.663921 2013 ts_tablet_manager.cc:1397] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 09:05:17.664345 2013 raft_consensus.cc:348] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.664462 2013 raft_consensus.cc:374] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:17.664505 2013 raft_consensus.cc:729] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb594f1faef4721b1b8f3e4941297aa, State: Initialized, Role: FOLLOWER
I20260620 09:05:17.664638 2013 consensus_queue.cc:260] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.664918 2013 ts_tablet_manager.cc:1428] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:17.665309 2023 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.665445 2023 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.755110 2020 raft_consensus.cc:482] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:17.755263 2020 raft_consensus.cc:504] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.755789 2020 leader_election.cc:288] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953), 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639)
I20260620 09:05:17.756197 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1823ae494f954cb89bc30ec5aab16996" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb594f1faef4721b1b8f3e4941297aa" is_pre_election: true
I20260620 09:05:17.756317 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1823ae494f954cb89bc30ec5aab16996" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72" is_pre_election: true
I20260620 09:05:17.756371 1804 raft_consensus.cc:2454] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 0.
I20260620 09:05:17.756479 1944 raft_consensus.cc:2454] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 0.
I20260620 09:05:17.756673 1596 leader_election.cc:302] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fb594f1faef4721b1b8f3e4941297aa, de6682c6a7fe4ac1b34dbd7576fd6877; no voters:
I20260620 09:05:17.756829 2020 raft_consensus.cc:2781] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:17.756886 2020 raft_consensus.cc:482] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:17.756924 2020 raft_consensus.cc:3037] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.758077 2020 raft_consensus.cc:504] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.758445 2020 leader_election.cc:288] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 election: Requested vote from peers 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953), 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639)
I20260620 09:05:17.758734 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1823ae494f954cb89bc30ec5aab16996" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb594f1faef4721b1b8f3e4941297aa"
I20260620 09:05:17.758864 1804 raft_consensus.cc:3037] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.758895 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1823ae494f954cb89bc30ec5aab16996" candidate_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "834bdc5131984d17979d9d266cd56f72"
I20260620 09:05:17.758993 1944 raft_consensus.cc:3037] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:17.759811 1944 raft_consensus.cc:2454] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 1.
I20260620 09:05:17.759976 1804 raft_consensus.cc:2454] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6682c6a7fe4ac1b34dbd7576fd6877 in term 1.
I20260620 09:05:17.760133 1596 leader_election.cc:302] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 834bdc5131984d17979d9d266cd56f72, de6682c6a7fe4ac1b34dbd7576fd6877; no voters:
I20260620 09:05:17.760322 2020 raft_consensus.cc:2781] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:17.760409 2020 raft_consensus.cc:686] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 LEADER]: Becoming Leader. State: Replica: de6682c6a7fe4ac1b34dbd7576fd6877, State: Running, Role: LEADER
I20260620 09:05:17.760533 2020 consensus_queue.cc:237] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } }
I20260620 09:05:17.761262 2044 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: TxnStatusTablet state changed. Reason: New leader de6682c6a7fe4ac1b34dbd7576fd6877. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.761372 2044 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:17.761592 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:05:17.761842 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:05:17.762674 1506 catalog_manager.cc:5495] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 reported cstate change: term changed from 0 to 1, leader changed from <none> to de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } health_report { overall_health: UNKNOWN } } }
I20260620 09:05:17.787340 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.783039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.98.1:41697)
0620 09:05:17.783290 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.783294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.783307 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:17.783412 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.783414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.783468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.783560 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:17.783846 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.784397 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.785694 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.786047 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.786166 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.786508 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.786957 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.787043 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.787150 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.787202 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260620 09:05:17.787367 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.782917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.0.1:33182)
0620 09:05:17.783149 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.783165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.783272 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.783596 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.783599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.783609 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:17.783720 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.783726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.784502 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.784505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.785549 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.785560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.786200 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.786204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.786293 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.786349 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.787047 (+ 698us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.787061 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.787133 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.787224 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 09:05:17.790707 1944 raft_consensus.cc:1261] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Refusing update from remote peer de6682c6a7fe4ac1b34dbd7576fd6877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.790870 1804 raft_consensus.cc:1261] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Refusing update from remote peer de6682c6a7fe4ac1b34dbd7576fd6877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.791275 2044 consensus_queue.cc:1035] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.791433 2020 consensus_queue.cc:1035] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.793203 2023 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: TxnStatusTablet state changed. Reason: New leader de6682c6a7fe4ac1b34dbd7576fd6877. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.793358 2023 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.793928 2021 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: TxnStatusTablet state changed. Reason: New leader de6682c6a7fe4ac1b34dbd7576fd6877. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.794051 2021 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.795468 2020 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.795581 2020 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:17.795681 2061 mvcc.cc:205] Tried to move back new op lower bound from 7298852117665193984 to 7298852117548720128. Current Snapshot: MvccSnapshot[applied={T|T < 7298852117665193984}]
I20260620 09:05:17.795873 2023 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.795995 2023 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 9fb594f1faef4721b1b8f3e4941297aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.798031 2044 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.798136 2044 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P de6682c6a7fe4ac1b34dbd7576fd6877: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:17.799250 2021 tablet_replica.cc:439] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de6682c6a7fe4ac1b34dbd7576fd6877" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 } } peers { permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 } } }
I20260620 09:05:17.799366 2021 tablet_replica.cc:442] T 1823ae494f954cb89bc30ec5aab16996 P 834bdc5131984d17979d9d266cd56f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:17.799341 2062 mvcc.cc:205] Tried to move back new op lower bound from 7298852117665193984 to 7298852117548720128. Current Snapshot: MvccSnapshot[applied={T|T < 7298852117665193984}]
I20260620 09:05:17.806116 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.802782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.1:41697)
0620 09:05:17.802915 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.802919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.803137 (+ 218us) server_negotiation.cc:407] Connection header received
0620 09:05:17.803308 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.803311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.803363 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.803440 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:17.803694 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.804205 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.805077 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.805305 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.805415 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.805671 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.805745 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.805829 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.805924 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.805973 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:05:17.816876 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.813575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57603 (local address 127.0.98.1:41697)
0620 09:05:17.813697 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.813701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.813893 (+ 192us) server_negotiation.cc:407] Connection header received
0620 09:05:17.814017 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.814019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.814059 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.814120 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:17.814419 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.814960 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.815831 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.816040 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.816152 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.816431 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.816502 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.816580 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.816666 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.816707 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:05:17.816883 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.813516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.98.1:57603)
0620 09:05:17.813806 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:17.813821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:17.813927 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:17.814160 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:17.814163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:17.814170 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:17.814289 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.814295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.815061 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.815067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.815709 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:17.815717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.816160 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.816163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:17.816256 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.816302 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:17.816583 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:17.816590 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:05:17.816650 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 09:05:17.816723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":108,"threads_started":1}
I20260620 09:05:17.824487 1804 raft_consensus.cc:1261] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Refusing update from remote peer de6682c6a7fe4ac1b34dbd7576fd6877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.824528 1944 raft_consensus.cc:1261] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Refusing update from remote peer de6682c6a7fe4ac1b34dbd7576fd6877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:17.824916 2044 consensus_queue.cc:1035] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834bdc5131984d17979d9d266cd56f72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.825150 2020 consensus_queue.cc:1035] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb594f1faef4721b1b8f3e4941297aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:17.868217 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:17.868398 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:17.870442 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8daf0dd6e3784840938a41d8f78bc076: Sending DeleteTablet for 3 replicas of tablet b122921b6fc54bd897603cd3f6bc25b6
I20260620 09:05:17.871131 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60164
I20260620 09:05:17.871179 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33172
I20260620 09:05:17.871186 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet b122921b6fc54bd897603cd3f6bc25b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53528
I20260620 09:05:17.871538 2080 tablet_replica.cc:330] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: stopping tablet replica
I20260620 09:05:17.871673 2080 raft_consensus.cc:2229] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:17.871795 2080 raft_consensus.cc:2258] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.872470 2080 ts_tablet_manager.cc:1905] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.872665 2079 tablet_replica.cc:330] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: stopping tablet replica
I20260620 09:05:17.872828 2079 raft_consensus.cc:2229] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:17.873167 2079 raft_consensus.cc:2258] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.873781 2079 ts_tablet_manager.cc:1905] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.875499 2080 ts_tablet_manager.cc:1918] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:17.875586 2080 log.cc:1196] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/b122921b6fc54bd897603cd3f6bc25b6
I20260620 09:05:17.875842 2080 ts_tablet_manager.cc:1939] T b122921b6fc54bd897603cd3f6bc25b6 P 9fb594f1faef4721b1b8f3e4941297aa: Deleting consensus metadata
I20260620 09:05:17.876027 2079 ts_tablet_manager.cc:1918] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:17.876116 2079 log.cc:1196] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/b122921b6fc54bd897603cd3f6bc25b6
I20260620 09:05:17.876343 2079 ts_tablet_manager.cc:1939] T b122921b6fc54bd897603cd3f6bc25b6 P de6682c6a7fe4ac1b34dbd7576fd6877: Deleting consensus metadata
I20260620 09:05:17.876541 1492 catalog_manager.cc:4841] TS 9fb594f1faef4721b1b8f3e4941297aa (127.0.98.2:41953): tablet b122921b6fc54bd897603cd3f6bc25b6 (table test-table [id=6724c8b5e9644fc8897534c078ec9cf6]) successfully deleted
I20260620 09:05:17.877059 2078 tablet_replica.cc:330] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: stopping tablet replica
I20260620 09:05:17.877022 1491 catalog_manager.cc:4841] TS de6682c6a7fe4ac1b34dbd7576fd6877 (127.0.98.1:41697): tablet b122921b6fc54bd897603cd3f6bc25b6 (table test-table [id=6724c8b5e9644fc8897534c078ec9cf6]) successfully deleted
I20260620 09:05:17.877194 2078 raft_consensus.cc:2229] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:17.877319 2078 raft_consensus.cc:2258] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:17.877965 2078 ts_tablet_manager.cc:1905] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:17.880393 2078 ts_tablet_manager.cc:1918] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:17.880470 2078 log.cc:1196] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/b122921b6fc54bd897603cd3f6bc25b6
I20260620 09:05:17.880725 2078 ts_tablet_manager.cc:1939] T b122921b6fc54bd897603cd3f6bc25b6 P 834bdc5131984d17979d9d266cd56f72: Deleting consensus metadata
I20260620 09:05:17.881531 1492 catalog_manager.cc:4841] TS 834bdc5131984d17979d9d266cd56f72 (127.0.98.3:33639): tablet b122921b6fc54bd897603cd3f6bc25b6 (table test-table [id=6724c8b5e9644fc8897534c078ec9cf6]) successfully deleted
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:17.887669 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.872910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.0.98.1:41697)
0620 09:05:17.873015 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.873020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.876789 (+ 3769us) server_negotiation.cc:407] Connection header received
0620 09:05:17.876963 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.876967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.877016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.877087 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:17.878314 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.879088 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.880291 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.880646 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.880759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.883963 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:17.883972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:17.883988 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:17.884023 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:17.885830 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.886329 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.886338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.886340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.886444 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.886718 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.886723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.886725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.886978 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:17.887196 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.887350 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.887475 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 09:05:17.888608 1700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33204
I20260620 09:05:17.898316 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:17.890412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.0.98.62:39469)
0620 09:05:17.890540 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:17.890544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:17.890617 (+ 73us) server_negotiation.cc:407] Connection header received
0620 09:05:17.890728 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:17.890731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:17.890785 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:17.890851 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:17.891647 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.892195 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.893329 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:17.893553 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:17.893631 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:17.895122 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:17.895125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:17.895127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:17.895153 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:17.896571 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.897182 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.897184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.897186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.897260 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:17.897534 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:17.897536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:17.897537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:17.897704 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:17.897812 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:17.898054 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 09:05:17.898153 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 09:05:17.898738 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39264
I20260620 09:05:17.900120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1579
I20260620 09:05:17.907807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1719
I20260620 09:05:17.914520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1859
I20260620 09:05:17.920116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 1475
2026-06-20T09:05:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3015 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[2087](info): set up 2 sockets
Jun 20 09:05:17 dist-test-slave-92t4 krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260620 09:05:18.998667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 09:05:19.199361 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.201s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:19.300158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:19.411957 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:19.412299 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:19.412372 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:19.416432 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:19.416531 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:05:19.416563 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:19.416627 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:19.416685 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:19.416723 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:19.422086 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:19.423285 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:19.424368 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:19.433002 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.433009 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.433009 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:19.433375 2100 server_base.cc:1034] running on GCE node
I20260620 09:05:19.434022 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:19.435138 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:19.436316 2100 hybrid_clock.cc:648] HybridClock initialized: now 1781946319436300 us; error 29 us; skew 500 ppm
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:19.439018 2100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:19.440120 2100 webserver.cc:460] Webserver started at http://127.0.98.62:36295/ using document root <none> and password file <none>
I20260620 09:05:19.440840 2100 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:19.440973 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:19.441211 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:19.443696 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "f2b0dad344be4dfb9879a00e146a1522"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "78322376a8068a63ded9a325014738ca"
server_key_iv: "3e5913cf87b34fd3bc6e6e070fb3bad8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.444283 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2b0dad344be4dfb9879a00e146a1522"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "78322376a8068a63ded9a325014738ca"
server_key_iv: "3e5913cf87b34fd3bc6e6e070fb3bad8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.448056 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:19.450582 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:19.451790 2100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:19.451907 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "f2b0dad344be4dfb9879a00e146a1522"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "78322376a8068a63ded9a325014738ca"
server_key_iv: "3e5913cf87b34fd3bc6e6e070fb3bad8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.452006 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:19.474399 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:19.477737 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:19.477950 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:19.485859 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39875
I20260620 09:05:19.485868 2177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39875 every 8 connection(s)
I20260620 09:05:19.486872 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:19.489784 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:19.495592 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Bootstrap starting.
I20260620 09:05:19.496915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2100
I20260620 09:05:19.497035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:19.497318 392 external_mini_cluster.cc:1428] Setting key 5218095c822ca049f4f3890f2b6d12e0
I20260620 09:05:19.498391 2178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:19.499159 2178 log.cc:824] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:19.501401 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: No bootstrap required, opened a new log
I20260620 09:05:19.506031 2178 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } }
I20260620 09:05:19.506309 2178 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:19.506383 2178 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b0dad344be4dfb9879a00e146a1522, State: Initialized, Role: FOLLOWER
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:19.506875 2178 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } }
I20260620 09:05:19.507016 2178 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:19.507081 2178 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:19.507179 2178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:19.508113 2178 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } }
I20260620 09:05:19.508508 2178 leader_election.cc:302] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b0dad344be4dfb9879a00e146a1522; no voters:
I20260620 09:05:19.508878 2178 leader_election.cc:288] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:19.508988 2183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:19.509277 2183 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [term 1 LEADER]: Becoming Leader. State: Replica: f2b0dad344be4dfb9879a00e146a1522, State: Running, Role: LEADER
I20260620 09:05:19.509624 2183 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } }
I20260620 09:05:19.510385 2178 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:19.510702 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b0dad344be4dfb9879a00e146a1522. Latest consensus state: current_term: 1 leader_uuid: "f2b0dad344be4dfb9879a00e146a1522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } } }
I20260620 09:05:19.510818 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:19.511269 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b0dad344be4dfb9879a00e146a1522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b0dad344be4dfb9879a00e146a1522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39875 } } }
I20260620 09:05:19.511410 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:19.513531 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:19.498456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.98.62:39875)
0620 09:05:19.498909 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:19.498925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:19.498960 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:19.499749 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:19.499772 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:19.500157 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:19.500446 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:19.501329 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.502354 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.503610 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.504103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.504221 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:19.507182 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:19.507189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:19.507209 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:19.507245 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:19.509366 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.509928 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.509934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.509941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.510079 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.510408 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.510411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.510414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.510953 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:19.511968 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:19.512162 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:05:19.512509 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
W20260620 09:05:19.515409 2198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:19.515499 2198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:19.515568 2187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:19.516321 2187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:19.522336 2187 catalog_manager.cc:1347] Generated new cluster ID: d0d00223d25f438884450e9db1e2d3c3
I20260620 09:05:19.522425 2187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:19.531124 2187 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:19.532560 2187 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:19.540568 2187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Generated new TSK 0
I20260620 09:05:19.541414 2187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:19.574221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39875
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:19.686971 2204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:19.687311 2204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:19.687400 2204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:19.691519 2204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:19.691612 2204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:19.691674 2204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:05:19.691712 2204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:19.691802 2204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:19.697269 2204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:19.698421 2204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:19.699491 2204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:19.709004 2210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.709025 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.709002 2209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:19.709313 2204 server_base.cc:1034] running on GCE node
I20260620 09:05:19.709785 2204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:19.710470 2204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:19.711668 2204 hybrid_clock.cc:648] HybridClock initialized: now 1781946319711650 us; error 26 us; skew 500 ppm
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:19.714669 2204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:19.715538 2204 webserver.cc:460] Webserver started at http://127.0.98.1:38923/ using document root <none> and password file <none>
I20260620 09:05:19.716077 2204 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:19.716135 2204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:19.716306 2204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:19.718405 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "f35a63a4022d4e6a8194a9802a688966"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "26fa47498bd4fc5896ae1ad961d8dd4e"
server_key_iv: "045e1768768583aff6e271f866a877d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.718937 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f35a63a4022d4e6a8194a9802a688966"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "26fa47498bd4fc5896ae1ad961d8dd4e"
server_key_iv: "045e1768768583aff6e271f866a877d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.722715 2204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:19.725163 2219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:19.726403 2204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:19.726501 2204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "f35a63a4022d4e6a8194a9802a688966"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "26fa47498bd4fc5896ae1ad961d8dd4e"
server_key_iv: "045e1768768583aff6e271f866a877d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.726635 2204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:19.758960 2204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:19.762568 2204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:19.762779 2204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:19.764302 2204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:19.764375 2204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:19.764449 2204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:19.764488 2204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:19.775735 2204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45141
I20260620 09:05:19.775849 2334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45141 every 8 connection(s)
I20260620 09:05:19.776788 2204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:19.778875 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:19.765034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55993 (local address 127.0.98.62:39875)
0620 09:05:19.765215 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:19.765222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:19.766692 (+ 1470us) server_negotiation.cc:407] Connection header received
0620 09:05:19.767688 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:19.767693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:19.767762 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:19.767841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:19.769622 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.770560 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.771897 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.772173 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.772294 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:19.775127 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:19.775131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:19.775133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:19.775168 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:19.776698 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.777277 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.777281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.777283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.777376 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.777775 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.777778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.777780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.777977 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:19.778094 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:19.778571 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 09:05:19.778701 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:05:19.779726 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:19.765591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.1:55993)
0620 09:05:19.766520 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:19.766573 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:19.767403 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:19.768336 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:19.768347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:19.768799 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:19.769475 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:19.769491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.770700 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.770706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:19.771767 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:19.771777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.772343 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.772347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:19.772535 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:19.773332 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:19.773348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:19.774943 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:19.776821 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:19.776826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:19.776841 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:19.777171 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:19.777482 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:19.777485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:19.777486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:19.777667 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:19.778081 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:19.778088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:19.778332 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 09:05:19.778514 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 09:05:19.778848 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":152,"threads_started":1}
I20260620 09:05:19.780938 2335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39875
I20260620 09:05:19.781111 2335 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:19.781432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2204
I20260620 09:05:19.781549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:19.781708 2335 heartbeater.cc:511] Master 127.0.98.62:39875 requested a full tablet report, sending...
I20260620 09:05:19.781849 392 external_mini_cluster.cc:1428] Setting key 0cd06d63a1fed672bc8430f34bf2f764
I20260620 09:05:19.783433 2132 ts_manager.cc:194] Registered new tserver with Master: f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141)
I20260620 09:05:19.785068 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55993
I20260620 09:05:19.790956 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:19.783735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48111 (local address 127.0.98.62:39875)
0620 09:05:19.783892 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:19.783895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:19.783904 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:05:19.783932 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:19.783934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:19.783966 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:19.784050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:19.784496 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.785153 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.786428 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.786647 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.786749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:19.787869 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:19.787873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:19.787876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:19.787899 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:19.789368 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.789807 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.789811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.789813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.789890 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:19.790157 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:19.790160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:19.790162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:19.790333 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:19.790431 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:19.790680 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 09:05:19.790795 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 09:05:19.790967 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:19.783376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.1:48111)
0620 09:05:19.783631 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:19.783649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:19.783742 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:19.784066 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:19.784068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:19.784279 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:19.784383 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:19.784388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.785308 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.785313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:19.786301 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:19.786313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:19.786756 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:19.786760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:19.786850 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:19.787378 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:19.787397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:19.787739 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:19.789496 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:19.789499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:19.789502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:19.789697 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:19.789975 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:19.789979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:19.789981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:19.790066 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:19.790432 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:19.790436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:19.790528 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 09:05:19.790628 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 09:05:19.790782 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:19.808239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39875
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:19.921725 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:19.922029 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:19.922077 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:19.926191 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:19.926258 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:19.926312 2344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:05:19.926345 2344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:19.926379 2344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:19.931922 2344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:19.933013 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:19.934121 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:19.943902 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.943986 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:19.943907 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:19.943981 2344 server_base.cc:1034] running on GCE node
I20260620 09:05:19.944536 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:19.945264 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:19.946439 2344 hybrid_clock.cc:648] HybridClock initialized: now 1781946319946414 us; error 36 us; skew 500 ppm
Jun 20 09:05:19 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:19.949231 2344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:19.950093 2344 webserver.cc:460] Webserver started at http://127.0.98.2:37603/ using document root <none> and password file <none>
I20260620 09:05:19.950695 2344 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:19.950783 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:19.951011 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:19.953155 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "2468acf0671bff5fc35bf8905e60daaa"
server_key_iv: "4111ece8b57a3a7f2e7da89ee7693521"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.953784 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "2468acf0671bff5fc35bf8905e60daaa"
server_key_iv: "4111ece8b57a3a7f2e7da89ee7693521"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.957541 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:19.959959 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:19.961265 2344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:19.961387 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
format_stamp: "Formatted at 2026-06-20 09:05:19 on dist-test-slave-92t4"
server_key: "2468acf0671bff5fc35bf8905e60daaa"
server_key_iv: "4111ece8b57a3a7f2e7da89ee7693521"
server_key_version: "encryptionkey@0"
I20260620 09:05:19.961493 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:19.998121 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:20.001691 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:20.001907 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:20.003439 2344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:20.003512 2344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:20.003583 2344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:20.003626 2344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:20.014729 2344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42919
I20260620 09:05:20.014796 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42919 every 8 connection(s)
I20260620 09:05:20.015877 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:20.018357 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.004353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55837 (local address 127.0.98.62:39875)
0620 09:05:20.004483 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.004487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.005653 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 09:05:20.006774 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.006781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.006855 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.006960 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.008648 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.009575 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.011005 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.011309 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.011454 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.014637 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.014640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.014642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.014687 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.016201 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.016804 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.016809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.016811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.016894 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.017265 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.017270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.017275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.017464 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.017567 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.018059 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.018183 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:20.019142 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.004978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.2:55837)
0620 09:05:20.005501 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.005546 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.006474 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.007196 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.007206 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.007711 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:20.008448 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.008472 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.009703 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.009708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.010872 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.010882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.011513 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.011521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.011762 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.012469 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:20.012493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:20.014437 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:20.016327 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.016333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.016349 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.016698 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.016992 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.016995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.016997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.017174 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.017574 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:20.017582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:20.017828 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.018004 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.018328 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":138,"threads_started":1}
I20260620 09:05:20.020282 2475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39875
I20260620 09:05:20.020430 2475 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:20.020912 2475 heartbeater.cc:511] Master 127.0.98.62:39875 requested a full tablet report, sending...
I20260620 09:05:20.021950 2131 ts_manager.cc:194] Registered new tserver with Master: 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:20.022667 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55837
I20260620 09:05:20.025301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2344
I20260620 09:05:20.025418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:20.025681 392 external_mini_cluster.cc:1428] Setting key 0e4286da4d31d575e971d2ba744af080
I20260620 09:05:20.029950 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.022454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.2:44961)
0620 09:05:20.022710 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.022735 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.022861 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.023130 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.023133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.023327 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:20.023423 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.023428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.024370 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.024372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.024910 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.024916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.025441 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.025460 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.025553 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.026070 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:20.026082 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:20.026465 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:20.028355 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.028357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.028359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.028539 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.028868 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.028873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.028876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.028965 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.029397 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:20.029400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:20.029497 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.029606 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.029770 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260620 09:05:20.030407 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.022511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44961 (local address 127.0.98.62:39875)
0620 09:05:20.022629 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.022634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.022799 (+ 165us) server_negotiation.cc:407] Connection header received
0620 09:05:20.022964 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.022967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.023016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.023086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.023528 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.024259 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.025002 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.025328 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.025441 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.026574 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.026577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.026579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.026599 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.028254 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.028628 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.028634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.028639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.028733 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.029077 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.029083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.029086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.029298 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.029408 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.030186 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.030303 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:20.050346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39875
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:20.166968 2484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:20.167230 2484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:20.167279 2484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:20.171288 2484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:20.171339 2484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:20.171378 2484 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 09:05:20.171392 2484 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:20.171423 2484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:20.176591 2484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:20.177625 2484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:20.178609 2484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:20.188081 2490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:20.188134 2492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:20.188095 2489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:20.188197 2484 server_base.cc:1034] running on GCE node
I20260620 09:05:20.188879 2484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:20.189579 2484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:20.190764 2484 hybrid_clock.cc:648] HybridClock initialized: now 1781946320190752 us; error 34 us; skew 500 ppm
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:20.193468 2484 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:20.194319 2484 webserver.cc:460] Webserver started at http://127.0.98.3:38005/ using document root <none> and password file <none>
I20260620 09:05:20.194887 2484 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:20.194977 2484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:20.195200 2484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:20.197376 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ff21e10975743beb479f6ec4e71b6ec"
format_stamp: "Formatted at 2026-06-20 09:05:20 on dist-test-slave-92t4"
server_key: "c206b7a4250e3384487c1be423129faa"
server_key_iv: "ba665953f0e2bec1df93f2656b44f969"
server_key_version: "encryptionkey@0"
I20260620 09:05:20.197938 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff21e10975743beb479f6ec4e71b6ec"
format_stamp: "Formatted at 2026-06-20 09:05:20 on dist-test-slave-92t4"
server_key: "c206b7a4250e3384487c1be423129faa"
server_key_iv: "ba665953f0e2bec1df93f2656b44f969"
server_key_version: "encryptionkey@0"
I20260620 09:05:20.201509 2484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:05:20.203828 2499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:20.205027 2484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:20.205199 2484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "6ff21e10975743beb479f6ec4e71b6ec"
format_stamp: "Formatted at 2026-06-20 09:05:20 on dist-test-slave-92t4"
server_key: "c206b7a4250e3384487c1be423129faa"
server_key_iv: "ba665953f0e2bec1df93f2656b44f969"
server_key_version: "encryptionkey@0"
I20260620 09:05:20.205302 2484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:20.223517 2484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:20.226715 2484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:20.226918 2484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:20.228439 2484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:20.228508 2484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:20.228579 2484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:20.228633 2484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:20.239986 2484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43667
I20260620 09:05:20.240113 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43667 every 8 connection(s)
I20260620 09:05:20.241016 2484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:20.242491 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.229173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56359 (local address 127.0.98.62:39875)
0620 09:05:20.229291 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.229295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.230547 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 09:05:20.231726 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.231733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.231907 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.231990 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.233658 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.234267 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.235519 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.235872 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.235986 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.238789 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.238795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.238800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.238825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.240433 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.240968 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.240971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.240972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.241084 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.241500 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.241503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.241505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.241753 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.241857 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.242280 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.242376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:05:20.243279 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.229770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.3:56359)
0620 09:05:20.230341 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.230400 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.231428 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.232376 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.232382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.232836 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:20.233503 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.233525 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.234389 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.234393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.235395 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.235404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.236030 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.236034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.236237 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.236878 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:20.236901 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:20.238606 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:20.240532 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.240537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.240551 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.240870 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.241202 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.241207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.241209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.241374 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.241849 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:20.241854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:20.242070 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.242220 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.242449 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":136,"threads_started":1}
I20260620 09:05:20.244529 2615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39875
I20260620 09:05:20.244676 2615 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:20.245222 2615 heartbeater.cc:511] Master 127.0.98.62:39875 requested a full tablet report, sending...
I20260620 09:05:20.246181 2132 ts_manager.cc:194] Registered new tserver with Master: 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667)
I20260620 09:05:20.246948 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56359
I20260620 09:05:20.247135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2484
I20260620 09:05:20.247253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:20.247602 392 external_mini_cluster.cc:1428] Setting key e82c9d8e0f2419ae625631ce0938b580
I20260620 09:05:20.249516 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:05:20.254527 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.246425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.98.3:57395)
0620 09:05:20.246719 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.246733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.246813 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.247485 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.247488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.247775 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:20.248105 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.248113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.248903 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.248906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.249721 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.249731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.250132 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.250136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.250238 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.250726 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:20.250747 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:20.251199 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:20.253035 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.253078 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.253080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.253271 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.253546 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.253548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.253549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.253615 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.254070 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:20.254076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:20.254173 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.254268 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.254369 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260620 09:05:20.254917 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.246524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57395 (local address 127.0.98.62:39875)
0620 09:05:20.247261 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.247265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.247277 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:20.247322 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.247325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.247361 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.247468 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.248199 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.248781 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.249822 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.250024 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.250104 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.251322 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.251325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.251327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.251346 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.252936 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.253360 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.253367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.253371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.253458 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.253701 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.253707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.253712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.253960 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.254071 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.254355 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.254476 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654}
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:20.275352 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.267493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.98.62:39875)
0620 09:05:20.267647 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.267651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.267662 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:20.267696 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.267698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.267732 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.267812 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.268469 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.268976 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.269834 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.270043 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.270148 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.272369 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.272377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.272379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.272400 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.273811 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.274217 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.274222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.274224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.274298 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.274558 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.274563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.274564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.274732 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.274864 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.275082 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.275188 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:20.278239 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:20.281329 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:20.294803 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.289240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43667 (local address 127.0.0.1:33470)
0620 09:05:20.289561 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.289607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.289830 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.291203 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.291206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.291218 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.291281 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.291287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292351 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.292357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.293524 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.293533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.294408 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.294411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.294488 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.294518 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.294602 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.294689 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260620 09:05:20.294857 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.290078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.98.3:43667)
0620 09:05:20.290781 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.290791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.290813 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:05:20.290887 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.290892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.291062 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.291412 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.291518 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292242 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.293680 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.294304 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.294414 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.294575 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.294626 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.294713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":125,"threads_started":1}
I20260620 09:05:20.295619 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.289959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.0.1:33842)
0620 09:05:20.290495 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.290508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.290603 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.291136 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.291142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.291173 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.291259 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.291266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292700 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.292703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.294179 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.294190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.295220 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.295223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.295325 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.295348 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.295425 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.295495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":128,"thread_start_us":91,"threads_started":1}
I20260620 09:05:20.295696 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.290360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.0.98.2:42919)
0620 09:05:20.290720 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.290730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.290747 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:05:20.290805 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.290808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.291002 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.291160 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.291373 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292574 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.294309 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.295126 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.295212 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.295318 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.295467 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.295559 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260620 09:05:20.296895 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.289334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.0.1:41854)
0620 09:05:20.289617 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.289632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.289742 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.290927 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.290933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.290956 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.291106 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.291117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292700 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.292703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.294180 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.294190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.296534 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.296538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.296622 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.296646 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.296717 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.296778 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":142,"thread_start_us":114,"threads_started":1}
I20260620 09:05:20.297000 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.290031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.98.1:45141)
0620 09:05:20.290417 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.290426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.290469 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:05:20.290550 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.290555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.290771 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.290934 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.291418 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.292557 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.295369 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.296400 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.296506 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.296633 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.296785 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.296879 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":66,"threads_started":1}
I20260620 09:05:20.298331 2409 tablet_service.cc:1467] Processing CreateTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe (DEFAULT_TABLE table=test-table [id=74868d4dbe36401e93b1178438c8efe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.298223 2549 tablet_service.cc:1467] Processing CreateTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe (DEFAULT_TABLE table=test-table [id=74868d4dbe36401e93b1178438c8efe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.299235 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14eb4a0080c4a19b4d227d47aeb12fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.299268 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14eb4a0080c4a19b4d227d47aeb12fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.299355 2269 tablet_service.cc:1467] Processing CreateTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe (DEFAULT_TABLE table=test-table [id=74868d4dbe36401e93b1178438c8efe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.300261 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14eb4a0080c4a19b4d227d47aeb12fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.305898 2638 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Bootstrap starting.
I20260620 09:05:20.308046 2638 tablet_bootstrap.cc:654] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.308828 2638 log.cc:824] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:20.309355 2639 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Bootstrap starting.
I20260620 09:05:20.309765 2640 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Bootstrap starting.
I20260620 09:05:20.311075 2638 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: No bootstrap required, opened a new log
I20260620 09:05:20.311299 2638 ts_tablet_manager.cc:1397] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:20.311651 2639 tablet_bootstrap.cc:654] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.311844 2640 tablet_bootstrap.cc:654] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.312470 2639 log.cc:824] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:20.312513 2640 log.cc:824] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:20.314357 2640 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: No bootstrap required, opened a new log
I20260620 09:05:20.314512 2639 tablet_bootstrap.cc:492] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: No bootstrap required, opened a new log
I20260620 09:05:20.314554 2640 ts_tablet_manager.cc:1397] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:05:20.314687 2639 ts_tablet_manager.cc:1397] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:05:20.314919 2638 raft_consensus.cc:348] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.315152 2638 raft_consensus.cc:374] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.315203 2638 raft_consensus.cc:729] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.315649 2638 consensus_queue.cc:260] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.316285 2475 heartbeater.cc:503] Master 127.0.98.62:39875 was elected leader, sending a full tablet report...
I20260620 09:05:20.316653 2638 ts_tablet_manager.cc:1428] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:05:20.318746 2640 raft_consensus.cc:348] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.319006 2640 raft_consensus.cc:374] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.319063 2640 raft_consensus.cc:729] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.319507 2640 consensus_queue.cc:260] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.319428 2639 raft_consensus.cc:348] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.319653 2639 raft_consensus.cc:374] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.319696 2639 raft_consensus.cc:729] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.320161 2639 consensus_queue.cc:260] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.320539 2640 ts_tablet_manager.cc:1428] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:20.320888 2335 heartbeater.cc:503] Master 127.0.98.62:39875 was elected leader, sending a full tablet report...
I20260620 09:05:20.321125 2639 ts_tablet_manager.cc:1428] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:20.321148 2615 heartbeater.cc:503] Master 127.0.98.62:39875 was elected leader, sending a full tablet report...
W20260620 09:05:20.492426 2616 tablet.cc:2369] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:05:20.517246 2476 tablet.cc:2369] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:20.520099 2645 raft_consensus.cc:482] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:20.520308 2645 raft_consensus.cc:504] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.521641 2645 leader_election.cc:288] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:20.525688 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.521988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41975 (local address 127.0.98.3:43667)
0620 09:05:20.522171 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.522177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.522256 (+ 79us) server_negotiation.cc:407] Connection header received
0620 09:05:20.522413 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.522418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.522476 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.522554 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.522902 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.523542 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.524666 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.525145 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.525234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.525314 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.525476 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.525553 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 09:05:20.525712 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.522374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59053 (local address 127.0.98.2:42919)
0620 09:05:20.522497 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.522502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.522638 (+ 136us) server_negotiation.cc:407] Connection header received
0620 09:05:20.522805 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.522809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.522868 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.522946 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.523206 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.523722 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.524748 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.525182 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.525297 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.525360 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.525535 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.525611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:05:20.525733 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.521887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43667 (local address 127.0.98.1:41975)
0620 09:05:20.522158 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.522171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.522307 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.522573 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.522576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.522593 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.522794 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.522801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.523652 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.523655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.524550 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.524559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.525253 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.525259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.525351 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.525380 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.525453 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.525537 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"spinlock_wait_cycles":4992,"thread_start_us":98,"threads_started":1}
I20260620 09:05:20.525746 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.522287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.98.1:59053)
0620 09:05:20.522548 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.522560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.522687 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.522963 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.522966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.522987 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.523100 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.523106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.523820 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.523822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.524625 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.524638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.525308 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.525311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.525403 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.525420 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.525511 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.525571 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":20,"spinlock_wait_cycles":74368,"thread_start_us":83,"threads_started":1}
I20260620 09:05:20.526409 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14eb4a0080c4a19b4d227d47aeb12fe" candidate_uuid: "f35a63a4022d4e6a8194a9802a688966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff21e10975743beb479f6ec4e71b6ec" is_pre_election: true
I20260620 09:05:20.526450 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14eb4a0080c4a19b4d227d47aeb12fe" candidate_uuid: "f35a63a4022d4e6a8194a9802a688966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" is_pre_election: true
I20260620 09:05:20.526733 2425 raft_consensus.cc:2454] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35a63a4022d4e6a8194a9802a688966 in term 0.
I20260620 09:05:20.526727 2569 raft_consensus.cc:2454] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35a63a4022d4e6a8194a9802a688966 in term 0.
I20260620 09:05:20.527215 2221 leader_election.cc:302] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:20.527474 2645 raft_consensus.cc:2781] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:20.527550 2645 raft_consensus.cc:482] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:20.527608 2645 raft_consensus.cc:3037] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:05:20.528151 2336 tablet.cc:2369] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:20.528693 2645 raft_consensus.cc:504] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.529031 2645 leader_election.cc:288] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [CANDIDATE]: Term 1 election: Requested vote from peers 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:20.529450 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14eb4a0080c4a19b4d227d47aeb12fe" candidate_uuid: "f35a63a4022d4e6a8194a9802a688966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff21e10975743beb479f6ec4e71b6ec"
I20260620 09:05:20.529558 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14eb4a0080c4a19b4d227d47aeb12fe" candidate_uuid: "f35a63a4022d4e6a8194a9802a688966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
I20260620 09:05:20.529592 2569 raft_consensus.cc:3037] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.529675 2425 raft_consensus.cc:3037] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.531014 2425 raft_consensus.cc:2454] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35a63a4022d4e6a8194a9802a688966 in term 1.
I20260620 09:05:20.531379 2221 leader_election.cc:302] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea3d5dd8ab74fa9b7514825c4fed357, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:20.531456 2569 raft_consensus.cc:2454] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35a63a4022d4e6a8194a9802a688966 in term 1.
I20260620 09:05:20.531577 2645 raft_consensus.cc:2781] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:20.531802 2645 raft_consensus.cc:686] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 1 LEADER]: Becoming Leader. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Running, Role: LEADER
I20260620 09:05:20.532128 2645 consensus_queue.cc:237] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } }
I20260620 09:05:20.535157 2132 catalog_manager.cc:5495] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 reported cstate change: term changed from 0 to 1, leader changed from <none> to f35a63a4022d4e6a8194a9802a688966 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f35a63a4022d4e6a8194a9802a688966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } health_report { overall_health: UNKNOWN } } }
Jun 20 09:05:20 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:20.603145 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.594834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39875 (local address 127.0.0.1:52164)
0620 09:05:20.595162 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.595180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.595314 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.595567 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.595570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.596022 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:20.596112 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.596118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.596848 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.596855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.597516 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.597524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.597943 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.597946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.598036 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.598726 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:20.598737 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:20.600116 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:20.601742 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.601749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.601755 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.602001 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.602227 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:20.602229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:20.602230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:20.602315 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:20.602623 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:20.602628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:20.602740 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.602828 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.602973 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260620 09:05:20.603153 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.594937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.98.62:39875)
0620 09:05:20.595090 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.595095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.595257 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:05:20.595413 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.595416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.595461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.595547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:20.596232 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.596745 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.597624 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.597836 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.597941 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.600266 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:20.600269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:20.600271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:20.600295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:20.601652 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.602082 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.602085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.602086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.602152 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:20.602387 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:20.602389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:20.602390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:20.602545 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:20.602615 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.602882 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.603028 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":45}
I20260620 09:05:20.607111 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:05:20.607894 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:20.612802 2409 tablet_service.cc:1467] Processing CreateTablet for tablet dfa264441486429ba2a37f18445ff43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e6f1bdac8c04f398a70b8b2d920038c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:20.612805 2549 tablet_service.cc:1467] Processing CreateTablet for tablet dfa264441486429ba2a37f18445ff43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e6f1bdac8c04f398a70b8b2d920038c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:20.612805 2269 tablet_service.cc:1467] Processing CreateTablet for tablet dfa264441486429ba2a37f18445ff43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e6f1bdac8c04f398a70b8b2d920038c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:20.613600 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa264441486429ba2a37f18445ff43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.613600 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa264441486429ba2a37f18445ff43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.613600 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa264441486429ba2a37f18445ff43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.616343 2638 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: Bootstrap starting.
I20260620 09:05:20.616431 2639 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: Bootstrap starting.
I20260620 09:05:20.616559 2640 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: Bootstrap starting.
I20260620 09:05:20.617190 2638 tablet_bootstrap.cc:654] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.617477 2639 tablet_bootstrap.cc:654] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.617811 2640 tablet_bootstrap.cc:654] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.618786 2639 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: No bootstrap required, opened a new log
I20260620 09:05:20.618877 2639 ts_tablet_manager.cc:1397] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 09:05:20.619055 2638 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: No bootstrap required, opened a new log
I20260620 09:05:20.619109 2640 tablet_bootstrap.cc:492] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: No bootstrap required, opened a new log
I20260620 09:05:20.619133 2638 ts_tablet_manager.cc:1397] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 09:05:20.619189 2640 ts_tablet_manager.cc:1397] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:05:20.619356 2639 raft_consensus.cc:348] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.619472 2639 raft_consensus.cc:374] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.619505 2639 raft_consensus.cc:729] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.619544 2638 raft_consensus.cc:348] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.619639 2638 raft_consensus.cc:374] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.619639 2639 consensus_queue.cc:260] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.619727 2638 raft_consensus.cc:729] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.619624 2640 raft_consensus.cc:348] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.619774 2640 raft_consensus.cc:374] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.619820 2640 raft_consensus.cc:729] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.619846 2638 consensus_queue.cc:260] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.619933 2639 ts_tablet_manager.cc:1428] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:20.619957 2640 consensus_queue.cc:260] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.620122 2638 ts_tablet_manager.cc:1428] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:20.620229 2645 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.620368 2645 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.620342 2646 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.620477 2646 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.620653 2644 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.620795 2644 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.620227 2640 ts_tablet_manager.cc:1428] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:20.653692 2644 raft_consensus.cc:482] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:20.653852 2644 raft_consensus.cc:504] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.655062 2644 leader_election.cc:288] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667)
I20260620 09:05:20.659173 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.655345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43609 (local address 127.0.98.1:45141)
0620 09:05:20.655539 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.655546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.655669 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:05:20.655853 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.655860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.655917 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.655972 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.656258 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.656957 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.658132 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.658639 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.658722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.658773 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.658977 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.659028 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 09:05:20.659241 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.655242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.98.2:43609)
0620 09:05:20.655559 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.655574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.655735 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.656004 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.656009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.656028 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.656164 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.656169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.657077 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.657080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.658018 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.658027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.658754 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.658758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.658857 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.658885 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.658966 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.659045 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 09:05:20.659569 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.655465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38183 (local address 127.0.98.3:43667)
0620 09:05:20.655580 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.655586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.655767 (+ 181us) server_negotiation.cc:407] Connection header received
0620 09:05:20.655904 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.655906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.655944 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.656032 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.656743 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.657344 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.658449 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.658991 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.659083 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.659138 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.659379 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.659456 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:05:20.659819 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa264441486429ba2a37f18445ff43f" candidate_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966" is_pre_election: true
I20260620 09:05:20.659945 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.655318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43667 (local address 127.0.98.2:38183)
0620 09:05:20.655670 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.655696 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.655799 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.656053 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.656058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.656075 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.656634 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.656643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.657452 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.657454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.658332 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.658341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.659101 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.659105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.659220 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.659248 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.659731 (+ 483us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.659800 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260620 09:05:20.660051 2287 raft_consensus.cc:2454] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea3d5dd8ab74fa9b7514825c4fed357 in term 0.
I20260620 09:05:20.660334 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa264441486429ba2a37f18445ff43f" candidate_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff21e10975743beb479f6ec4e71b6ec" is_pre_election: true
I20260620 09:05:20.660482 2569 raft_consensus.cc:2454] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea3d5dd8ab74fa9b7514825c4fed357 in term 0.
I20260620 09:05:20.660553 2363 leader_election.cc:302] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea3d5dd8ab74fa9b7514825c4fed357, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:20.660899 2644 raft_consensus.cc:2781] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:20.660964 2644 raft_consensus.cc:482] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:20.661001 2644 raft_consensus.cc:3037] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.662016 2644 raft_consensus.cc:504] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.662348 2644 leader_election.cc:288] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [CANDIDATE]: Term 1 election: Requested vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667)
I20260620 09:05:20.662647 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa264441486429ba2a37f18445ff43f" candidate_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966"
I20260620 09:05:20.662755 2287 raft_consensus.cc:3037] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.662753 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa264441486429ba2a37f18445ff43f" candidate_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff21e10975743beb479f6ec4e71b6ec"
I20260620 09:05:20.662855 2569 raft_consensus.cc:3037] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.663812 2569 raft_consensus.cc:2454] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea3d5dd8ab74fa9b7514825c4fed357 in term 1.
I20260620 09:05:20.663925 2287 raft_consensus.cc:2454] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea3d5dd8ab74fa9b7514825c4fed357 in term 1.
I20260620 09:05:20.664100 2361 leader_election.cc:302] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, 8ea3d5dd8ab74fa9b7514825c4fed357; no voters:
I20260620 09:05:20.664294 2644 raft_consensus.cc:2781] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:20.664505 2644 raft_consensus.cc:686] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Running, Role: LEADER
I20260620 09:05:20.664861 2644 consensus_queue.cc:237] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.667057 2662 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: TxnStatusTablet state changed. Reason: New leader 8ea3d5dd8ab74fa9b7514825c4fed357. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.667201 2662 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:20.667471 2664 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:05:20.667680 2664 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:05:20.668293 2130 catalog_manager.cc:5495] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } health_report { overall_health: UNKNOWN } } }
I20260620 09:05:20.686012 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.682281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.2:42919)
0620 09:05:20.682419 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.682422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.682435 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:20.682607 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.682609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.682660 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.682741 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.682985 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.683478 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.684656 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.684858 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.684930 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.685272 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.685666 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.685756 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.685845 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.685889 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:05:20.686064 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.682196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.0.1:33848)
0620 09:05:20.682351 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.682368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.682471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.682768 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.682772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.682779 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:20.682888 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.682895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.683572 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.683575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.684544 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.684554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.684951 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.684957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.685087 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.685138 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.685758 (+ 620us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.685770 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.685840 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.685923 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 09:05:20.691511 2287 raft_consensus.cc:1261] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Refusing update from remote peer 8ea3d5dd8ab74fa9b7514825c4fed357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.691613 2569 raft_consensus.cc:1261] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Refusing update from remote peer 8ea3d5dd8ab74fa9b7514825c4fed357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.692149 2662 consensus_queue.cc:1035] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.692397 2644 consensus_queue.cc:1035] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.695621 2645 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: TxnStatusTablet state changed. Reason: New leader 8ea3d5dd8ab74fa9b7514825c4fed357. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.695744 2645 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.696990 2646 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: TxnStatusTablet state changed. Reason: New leader 8ea3d5dd8ab74fa9b7514825c4fed357. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.697131 2646 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.699569 2644 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.699692 2644 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:20.701432 2645 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.701550 2645 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P f35a63a4022d4e6a8194a9802a688966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.703341 2662 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.703450 2662 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 8ea3d5dd8ab74fa9b7514825c4fed357: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:20.705289 2646 tablet_replica.cc:439] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } } }
I20260620 09:05:20.705405 2646 tablet_replica.cc:442] T dfa264441486429ba2a37f18445ff43f P 6ff21e10975743beb479f6ec4e71b6ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:20.711068 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.707258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.98.1:45141)
0620 09:05:20.707380 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.707385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.707461 (+ 76us) server_negotiation.cc:407] Connection header received
0620 09:05:20.707609 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.707612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.707671 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.707770 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.708000 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.708567 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.709612 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.709874 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.709999 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.710267 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.710685 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.710812 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.710876 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.710933 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:05:20.722102 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.718637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47935 (local address 127.0.98.2:42919)
0620 09:05:20.718783 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.718788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.718891 (+ 103us) server_negotiation.cc:407] Connection header received
0620 09:05:20.719041 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.719044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.719098 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.719174 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.719444 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.720013 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.720874 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.721273 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.721373 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.721723 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.721788 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.721841 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.721958 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.721998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:05:20.722190 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.718533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.98.1:47935)
0620 09:05:20.718809 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.718828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.718945 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.719183 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.719185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.719193 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:20.719343 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.719348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.720119 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.720122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.720760 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.720768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.721392 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.721397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.721503 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.721555 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.721871 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.721877 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.721944 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.722024 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260620 09:05:20.729231 2569 raft_consensus.cc:1261] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Refusing update from remote peer f35a63a4022d4e6a8194a9802a688966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.729386 2425 raft_consensus.cc:1261] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Refusing update from remote peer f35a63a4022d4e6a8194a9802a688966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.729737 2649 consensus_queue.cc:1035] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.729895 2645 consensus_queue.cc:1035] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.735263 2669 mvcc.cc:205] Tried to move back new op lower bound from 7298852129702522880 to 7298852128901963776. Current Snapshot: MvccSnapshot[applied={T|T < 7298852129702522880}]
I20260620 09:05:20.739288 2667 mvcc.cc:205] Tried to move back new op lower bound from 7298852129702522880 to 7298852128901963776. Current Snapshot: MvccSnapshot[applied={T|T < 7298852129702522880}]
I20260620 09:05:20.753578 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.749789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59127 (local address 127.0.98.1:45141)
0620 09:05:20.749916 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.749922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.750140 (+ 218us) server_negotiation.cc:407] Connection header received
0620 09:05:20.750313 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.750316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.750364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.750416 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.750847 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.751534 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.752492 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.752723 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.752825 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.753142 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.753228 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.753290 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.753378 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.753442 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:20.753593 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.749706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.98.2:59127)
0620 09:05:20.750064 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.750075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.750188 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.750555 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.750558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.750579 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:20.750718 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.750726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.751662 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.751664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.752383 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.752391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.752820 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.752824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.752937 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.752976 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.753303 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.753310 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.753369 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.753445 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260620 09:05:20.778545 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:20.778764 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:20.781420 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Sending DeleteTablet for 3 replicas of tablet d14eb4a0080c4a19b4d227d47aeb12fe
I20260620 09:05:20.782213 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33470
I20260620 09:05:20.782275 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260620 09:05:20.783106 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet d14eb4a0080c4a19b4d227d47aeb12fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33842
I20260620 09:05:20.783565 2689 tablet_replica.cc:330] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: stopping tablet replica
I20260620 09:05:20.783833 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:05:20.783922 2689 raft_consensus.cc:2229] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 09:05:20.784250 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:20.784408 2689 raft_consensus.cc:2258] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:20.788841 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4198d4f53e4983861d25c820765de6 (DEFAULT_TABLE table=test-table [id=be9df34bcd274004b03895ff074027de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.789168 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4198d4f53e4983861d25c820765de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.789616 2549 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4198d4f53e4983861d25c820765de6 (DEFAULT_TABLE table=test-table [id=be9df34bcd274004b03895ff074027de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.789865 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4198d4f53e4983861d25c820765de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.790194 2689 ts_tablet_manager.cc:1905] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:20.792171 2639 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Bootstrap starting.
I20260620 09:05:20.793171 2639 tablet_bootstrap.cc:654] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.793152 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4198d4f53e4983861d25c820765de6 (DEFAULT_TABLE table=test-table [id=be9df34bcd274004b03895ff074027de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:20.793390 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4198d4f53e4983861d25c820765de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:20.794046 2640 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Bootstrap starting.
I20260620 09:05:20.794989 2640 tablet_bootstrap.cc:654] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.796396 2638 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Bootstrap starting.
I20260620 09:05:20.797339 2638 tablet_bootstrap.cc:654] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:20.801225 2687 tablet_replica.cc:330] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: stopping tablet replica
I20260620 09:05:20.801564 2687 raft_consensus.cc:2229] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:20.801813 2687 raft_consensus.cc:2258] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:20.802424 2689 ts_tablet_manager.cc:1918] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:20.802523 2689 log.cc:1196] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/d14eb4a0080c4a19b4d227d47aeb12fe
I20260620 09:05:20.802842 2689 ts_tablet_manager.cc:1939] T d14eb4a0080c4a19b4d227d47aeb12fe P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting consensus metadata
I20260620 09:05:20.804154 2117 catalog_manager.cc:4841] TS 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919): tablet d14eb4a0080c4a19b4d227d47aeb12fe (table test-table [id=74868d4dbe36401e93b1178438c8efe5]) successfully deleted
I20260620 09:05:20.804275 2687 ts_tablet_manager.cc:1905] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:20.805156 2640 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: No bootstrap required, opened a new log
I20260620 09:05:20.805254 2640 ts_tablet_manager.cc:1397] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260620 09:05:20.805655 2640 raft_consensus.cc:348] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.805790 2640 raft_consensus.cc:374] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.805831 2640 raft_consensus.cc:729] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.805951 2640 consensus_queue.cc:260] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.806211 2640 ts_tablet_manager.cc:1428] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:20.807246 2639 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: No bootstrap required, opened a new log
I20260620 09:05:20.807350 2639 ts_tablet_manager.cc:1397] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20260620 09:05:20.807385 2687 ts_tablet_manager.cc:1918] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:20.807477 2687 log.cc:1196] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/d14eb4a0080c4a19b4d227d47aeb12fe
I20260620 09:05:20.807847 2639 raft_consensus.cc:348] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.807986 2639 raft_consensus.cc:374] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.808017 2639 raft_consensus.cc:729] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.808144 2639 consensus_queue.cc:260] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.808410 2639 ts_tablet_manager.cc:1428] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:20.811161 2688 tablet_replica.cc:330] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: stopping tablet replica
I20260620 09:05:20.811611 2688 raft_consensus.cc:2229] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:20.812095 2688 raft_consensus.cc:2258] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:20.813083 2638 tablet_bootstrap.cc:492] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: No bootstrap required, opened a new log
I20260620 09:05:20.813176 2638 ts_tablet_manager.cc:1397] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20260620 09:05:20.813617 2638 raft_consensus.cc:348] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.813727 2638 raft_consensus.cc:374] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:20.813758 2638 raft_consensus.cc:729] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Initialized, Role: FOLLOWER
I20260620 09:05:20.813903 2688 ts_tablet_manager.cc:1905] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:20.813886 2638 consensus_queue.cc:260] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.814134 2638 ts_tablet_manager.cc:1428] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:20.807842 2687 ts_tablet_manager.cc:1939] T d14eb4a0080c4a19b4d227d47aeb12fe P 6ff21e10975743beb479f6ec4e71b6ec: Deleting consensus metadata
I20260620 09:05:20.818969 2688 ts_tablet_manager.cc:1918] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:20.819031 2117 catalog_manager.cc:4841] TS 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667): tablet d14eb4a0080c4a19b4d227d47aeb12fe (table test-table [id=74868d4dbe36401e93b1178438c8efe5]) successfully deleted
I20260620 09:05:20.819072 2688 log.cc:1196] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/d14eb4a0080c4a19b4d227d47aeb12fe
I20260620 09:05:20.819377 2688 ts_tablet_manager.cc:1939] T d14eb4a0080c4a19b4d227d47aeb12fe P f35a63a4022d4e6a8194a9802a688966: Deleting consensus metadata
I20260620 09:05:20.820461 2119 catalog_manager.cc:4841] TS f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141): tablet d14eb4a0080c4a19b4d227d47aeb12fe (table test-table [id=74868d4dbe36401e93b1178438c8efe5]) successfully deleted
I20260620 09:05:20.853564 2646 raft_consensus.cc:482] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:20.853704 2646 raft_consensus.cc:504] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.854967 2646 leader_election.cc:288] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:20.859356 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.855385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55821 (local address 127.0.98.1:45141)
0620 09:05:20.855544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.855549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.855626 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:05:20.855899 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.855903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.855951 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.855995 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.856391 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.857188 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.858342 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.858749 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.858897 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.858961 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.859164 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.859235 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 09:05:20.859381 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.855187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.98.3:55821)
0620 09:05:20.855541 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.855557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.855773 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.856127 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.856129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.856144 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.856273 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.856279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.857315 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.857318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.858199 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.858208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.858942 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.858945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.859044 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.859068 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.859138 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.859207 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260620 09:05:20.859759 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4198d4f53e4983861d25c820765de6" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966" is_pre_election: true
I20260620 09:05:20.859921 2287 raft_consensus.cc:2454] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:20.860069 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.855449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59585 (local address 127.0.98.2:42919)
0620 09:05:20.855560 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.855567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.856026 (+ 459us) server_negotiation.cc:407] Connection header received
0620 09:05:20.856221 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.856224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.856277 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.856329 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:20.856760 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.857373 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.858891 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.859394 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.859493 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.859560 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.859884 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.859963 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:05:20.860291 2503 leader_election.cc:302] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:20.860430 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.855190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.98.3:59585)
0620 09:05:20.855939 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.855952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.856100 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.856469 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.856471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.856487 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:20.856629 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.856635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.857487 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.857490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.858730 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.858739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.859646 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.859652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.859745 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.859767 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.860198 (+ 431us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.860280 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":395,"thread_start_us":68,"threads_started":1}
I20260620 09:05:20.860600 2646 raft_consensus.cc:2781] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:20.860654 2646 raft_consensus.cc:482] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:20.860699 2646 raft_consensus.cc:3037] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.860839 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4198d4f53e4983861d25c820765de6" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" is_pre_election: true
I20260620 09:05:20.860972 2425 raft_consensus.cc:2454] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:20.861910 2646 raft_consensus.cc:504] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.862254 2646 leader_election.cc:288] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Requested vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:20.862556 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4198d4f53e4983861d25c820765de6" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966"
I20260620 09:05:20.862619 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4198d4f53e4983861d25c820765de6" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
I20260620 09:05:20.862684 2287 raft_consensus.cc:3037] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.862732 2425 raft_consensus.cc:3037] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:20.863552 2287 raft_consensus.cc:2454] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:20.863816 2503 leader_election.cc:302] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:20.863969 2425 raft_consensus.cc:2454] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:20.863999 2646 raft_consensus.cc:2781] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:20.864192 2646 raft_consensus.cc:686] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Becoming Leader. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Running, Role: LEADER
I20260620 09:05:20.864445 2646 consensus_queue.cc:237] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:20.867041 2132 catalog_manager.cc:5495] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6ff21e10975743beb479f6ec4e71b6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:20.884178 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.880635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.0.98.3:43667)
0620 09:05:20.880736 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.880740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.880780 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:05:20.880968 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.880971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.881019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.881157 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.881372 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.881909 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.882713 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.882926 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.883028 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.883277 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.883805 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.883934 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.883974 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.884025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:05:20.894837 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.891195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36821 (local address 127.0.98.2:42919)
0620 09:05:20.891317 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.891320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.891535 (+ 215us) server_negotiation.cc:407] Connection header received
0620 09:05:20.891707 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.891710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.891762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.891840 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.892419 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.892897 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.893767 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.894001 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.894113 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.894366 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.894432 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.894526 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.894641 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.894694 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:05:20.894853 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.891096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.98.3:36821)
0620 09:05:20.891448 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.891460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.891575 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.891851 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.891854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.891861 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:20.892290 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.892296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.892999 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.893004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.893628 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.893637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.894105 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.894109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.894198 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.894236 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.894531 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.894539 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.894629 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.894704 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260620 09:05:20.900699 2287 raft_consensus.cc:1261] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.900709 2425 raft_consensus.cc:1261] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:20.901139 2695 consensus_queue.cc:1035] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.901304 2646 consensus_queue.cc:1035] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:20.921177 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.917410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43667 (local address 127.0.98.2:46049)
0620 09:05:20.917556 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:20.917572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:20.917653 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:20.918070 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:20.918073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:20.918080 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:20.918184 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.918193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.919103 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.919107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.919831 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:20.919841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.920285 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.920289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:20.920399 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.920427 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.920875 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.920884 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:05:20.920947 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 09:05:20.921032 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 09:05:20.921177 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:20.917790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46049 (local address 127.0.98.3:43667)
0620 09:05:20.917888 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:20.917891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:20.917902 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:20.917934 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:20.917936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:20.917972 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:20.918058 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:20.918305 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.918994 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.919950 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:20.920186 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:20.920289 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:20.920682 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:20.920763 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:20.920834 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:20.920959 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 09:05:20.921008 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 09:05:20.921948 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:05:22.484612 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:22.484800 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:22.487128 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Sending DeleteTablet for 3 replicas of tablet 9b4198d4f53e4983861d25c820765de6
I20260620 09:05:22.487773 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4198d4f53e4983861d25c820765de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260620 09:05:22.487766 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4198d4f53e4983861d25c820765de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33470
I20260620 09:05:22.488055 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b4198d4f53e4983861d25c820765de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33842
I20260620 09:05:22.488348 2727 tablet_replica.cc:330] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: stopping tablet replica
I20260620 09:05:22.488478 2727 raft_consensus.cc:2229] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:22.488579 2726 tablet_replica.cc:330] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: stopping tablet replica
I20260620 09:05:22.488602 2727 raft_consensus.cc:2258] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.488766 2726 raft_consensus.cc:2229] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:22.488921 2726 raft_consensus.cc:2258] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.489319 2727 ts_tablet_manager.cc:1905] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:22.489611 2726 ts_tablet_manager.cc:1905] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:22.492107 2727 ts_tablet_manager.cc:1918] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:05:22.492192 2727 log.cc:1196] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/9b4198d4f53e4983861d25c820765de6
I20260620 09:05:22.492319 2726 ts_tablet_manager.cc:1918] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:05:22.492429 2726 log.cc:1196] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/9b4198d4f53e4983861d25c820765de6
I20260620 09:05:22.492463 2727 ts_tablet_manager.cc:1939] T 9b4198d4f53e4983861d25c820765de6 P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting consensus metadata
I20260620 09:05:22.492734 2726 ts_tablet_manager.cc:1939] T 9b4198d4f53e4983861d25c820765de6 P f35a63a4022d4e6a8194a9802a688966: Deleting consensus metadata
I20260620 09:05:22.493224 2117 catalog_manager.cc:4841] TS 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919): tablet 9b4198d4f53e4983861d25c820765de6 (table test-table [id=be9df34bcd274004b03895ff074027de]) successfully deleted
I20260620 09:05:22.493497 2119 catalog_manager.cc:4841] TS f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141): tablet 9b4198d4f53e4983861d25c820765de6 (table test-table [id=be9df34bcd274004b03895ff074027de]) successfully deleted
I20260620 09:05:22.493815 2725 tablet_replica.cc:330] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: stopping tablet replica
I20260620 09:05:22.493959 2725 raft_consensus.cc:2229] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:22.494266 2725 raft_consensus.cc:2258] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.494799 2725 ts_tablet_manager.cc:1905] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:22.497459 2725 ts_tablet_manager.cc:1918] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:05:22.497551 2725 log.cc:1196] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/9b4198d4f53e4983861d25c820765de6
I20260620 09:05:22.497797 2725 ts_tablet_manager.cc:1939] T 9b4198d4f53e4983861d25c820765de6 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting consensus metadata
I20260620 09:05:22.498436 2117 catalog_manager.cc:4841] TS 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667): tablet 9b4198d4f53e4983861d25c820765de6 (table test-table [id=be9df34bcd274004b03895ff074027de]) successfully deleted
Jun 20 09:05:22 dist-test-slave-92t4 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:05:22 dist-test-slave-92t4 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:22.517499 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:22.508867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.0.98.62:39875)
0620 09:05:22.509174 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:22.509180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:22.509202 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:05:22.509333 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:22.509336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:22.509381 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:22.509459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:22.510190 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:22.510775 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:22.511727 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:22.511994 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:22.512073 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:22.514599 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:22.514602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:22.514604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:22.514626 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:22.516094 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:22.516432 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:22.516434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:22.516436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:22.516505 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:22.516709 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:22.516711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:22.516712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:22.516875 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:22.516973 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:22.517270 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 09:05:22.517368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260620 09:05:22.520015 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:22.520434 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:22.524922 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e (DEFAULT_TABLE table=test-table [id=8b626aa63a984dc69731a0f9223d38fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.525022 2549 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e (DEFAULT_TABLE table=test-table [id=8b626aa63a984dc69731a0f9223d38fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.524995 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e (DEFAULT_TABLE table=test-table [id=8b626aa63a984dc69731a0f9223d38fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.525279 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd8094ec5204e1ca4a1d735517e2b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.525277 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd8094ec5204e1ca4a1d735517e2b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.525378 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd8094ec5204e1ca4a1d735517e2b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.527371 2736 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Bootstrap starting.
I20260620 09:05:22.527805 2738 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Bootstrap starting.
I20260620 09:05:22.528369 2736 tablet_bootstrap.cc:654] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.528666 2738 tablet_bootstrap.cc:654] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.529232 2737 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Bootstrap starting.
I20260620 09:05:22.529739 2738 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: No bootstrap required, opened a new log
I20260620 09:05:22.529826 2738 ts_tablet_manager.cc:1397] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:05:22.530032 2737 tablet_bootstrap.cc:654] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.530242 2738 raft_consensus.cc:348] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.530367 2738 raft_consensus.cc:374] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.530402 2738 raft_consensus.cc:729] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.530553 2738 consensus_queue.cc:260] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.530922 2736 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: No bootstrap required, opened a new log
I20260620 09:05:22.530949 2738 ts_tablet_manager.cc:1428] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:22.531011 2736 ts_tablet_manager.cc:1397] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 09:05:22.531370 2737 tablet_bootstrap.cc:492] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: No bootstrap required, opened a new log
I20260620 09:05:22.531456 2737 ts_tablet_manager.cc:1397] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:22.531423 2736 raft_consensus.cc:348] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.531525 2736 raft_consensus.cc:374] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.531567 2736 raft_consensus.cc:729] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.531702 2736 consensus_queue.cc:260] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.531956 2736 ts_tablet_manager.cc:1428] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:22.532332 2737 raft_consensus.cc:348] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.532469 2737 raft_consensus.cc:374] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.532512 2737 raft_consensus.cc:729] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.532644 2737 consensus_queue.cc:260] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.532914 2737 ts_tablet_manager.cc:1428] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:22.632155 2719 raft_consensus.cc:482] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:22.632314 2719 raft_consensus.cc:504] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.632812 2719 leader_election.cc:288] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:22.633270 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd8094ec5204e1ca4a1d735517e2b9e" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966" is_pre_election: true
I20260620 09:05:22.633303 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd8094ec5204e1ca4a1d735517e2b9e" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" is_pre_election: true
I20260620 09:05:22.633430 2287 raft_consensus.cc:2454] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:22.633450 2425 raft_consensus.cc:2454] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:22.633764 2503 leader_election.cc:302] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:22.633947 2719 raft_consensus.cc:2781] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:22.634002 2719 raft_consensus.cc:482] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:22.634029 2719 raft_consensus.cc:3037] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.635123 2719 raft_consensus.cc:504] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.635417 2719 leader_election.cc:288] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Requested vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:22.635747 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd8094ec5204e1ca4a1d735517e2b9e" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966"
I20260620 09:05:22.635741 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd8094ec5204e1ca4a1d735517e2b9e" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
I20260620 09:05:22.635859 2425 raft_consensus.cc:3037] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.635857 2287 raft_consensus.cc:3037] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.636965 2287 raft_consensus.cc:2454] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:22.636965 2425 raft_consensus.cc:2454] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:22.637257 2503 leader_election.cc:302] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:22.637404 2719 raft_consensus.cc:2781] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:22.637490 2719 raft_consensus.cc:686] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Becoming Leader. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Running, Role: LEADER
I20260620 09:05:22.637621 2719 consensus_queue.cc:237] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.639196 2132 catalog_manager.cc:5495] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6ff21e10975743beb479f6ec4e71b6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:22.659957 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:22.657131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.0.98.3:43667)
0620 09:05:22.657381 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:22.657387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:22.657398 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:22.657452 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:22.657454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:22.657493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:22.657554 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:22.657791 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:22.658297 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:22.659059 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:22.659260 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:22.659353 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:22.659545 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:22.659611 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:22.659661 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:22.659732 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:05:22.659773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
I20260620 09:05:22.665786 2425 raft_consensus.cc:1261] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:22.665789 2287 raft_consensus.cc:1261] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:22.666181 2719 consensus_queue.cc:1035] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:22.666321 2646 consensus_queue.cc:1035] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:22.692035 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:22.692176 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:22.693885 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Sending DeleteTablet for 3 replicas of tablet 9cd8094ec5204e1ca4a1d735517e2b9e
I20260620 09:05:22.694330 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260620 09:05:22.694479 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33470
I20260620 09:05:22.694530 2726 tablet_replica.cc:330] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: stopping tablet replica
I20260620 09:05:22.694628 2725 tablet_replica.cc:330] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: stopping tablet replica
I20260620 09:05:22.694674 2726 raft_consensus.cc:2229] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:22.694746 2725 raft_consensus.cc:2229] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:22.694790 2726 raft_consensus.cc:2258] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.694912 2725 raft_consensus.cc:2258] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.694950 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd8094ec5204e1ca4a1d735517e2b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33842
I20260620 09:05:22.695107 2727 tablet_replica.cc:330] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: stopping tablet replica
I20260620 09:05:22.695223 2727 raft_consensus.cc:2229] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:22.695331 2727 raft_consensus.cc:2258] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:22.695596 2725 ts_tablet_manager.cc:1905] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:22.695596 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:05:22.695940 2727 ts_tablet_manager.cc:1905] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:05:22.696007 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:22.697154 2726 ts_tablet_manager.cc:1905] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:22.698243 2725 ts_tablet_manager.cc:1918] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:22.698323 2725 log.cc:1196] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/9cd8094ec5204e1ca4a1d735517e2b9e
I20260620 09:05:22.698484 2727 ts_tablet_manager.cc:1918] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:22.698551 2727 log.cc:1196] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/9cd8094ec5204e1ca4a1d735517e2b9e
I20260620 09:05:22.698575 2725 ts_tablet_manager.cc:1939] T 9cd8094ec5204e1ca4a1d735517e2b9e P 6ff21e10975743beb479f6ec4e71b6ec: Deleting consensus metadata
I20260620 09:05:22.698805 2727 ts_tablet_manager.cc:1939] T 9cd8094ec5204e1ca4a1d735517e2b9e P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting consensus metadata
I20260620 09:05:22.699551 2726 ts_tablet_manager.cc:1918] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:22.699584 2117 catalog_manager.cc:4841] TS 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919): tablet 9cd8094ec5204e1ca4a1d735517e2b9e (table test-table [id=8b626aa63a984dc69731a0f9223d38fb]) successfully deleted
I20260620 09:05:22.699636 2726 log.cc:1196] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/9cd8094ec5204e1ca4a1d735517e2b9e
I20260620 09:05:22.699735 2117 catalog_manager.cc:4841] TS 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667): tablet 9cd8094ec5204e1ca4a1d735517e2b9e (table test-table [id=8b626aa63a984dc69731a0f9223d38fb]) successfully deleted
I20260620 09:05:22.699904 2726 ts_tablet_manager.cc:1939] T 9cd8094ec5204e1ca4a1d735517e2b9e P f35a63a4022d4e6a8194a9802a688966: Deleting consensus metadata
I20260620 09:05:22.700520 2119 catalog_manager.cc:4841] TS f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141): tablet 9cd8094ec5204e1ca4a1d735517e2b9e (table test-table [id=8b626aa63a984dc69731a0f9223d38fb]) successfully deleted
I20260620 09:05:22.701073 2409 tablet_service.cc:1467] Processing CreateTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 (DEFAULT_TABLE table=test-table [id=77886894a99244e79d04184a1060d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.701083 2269 tablet_service.cc:1467] Processing CreateTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 (DEFAULT_TABLE table=test-table [id=77886894a99244e79d04184a1060d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.701321 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cf9066bd7c409da887193e3d8c9dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.701331 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cf9066bd7c409da887193e3d8c9dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.701365 2549 tablet_service.cc:1467] Processing CreateTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 (DEFAULT_TABLE table=test-table [id=77886894a99244e79d04184a1060d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:22.701573 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cf9066bd7c409da887193e3d8c9dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:22.703428 2736 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: Bootstrap starting.
I20260620 09:05:22.703584 2737 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Bootstrap starting.
I20260620 09:05:22.703640 2738 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: Bootstrap starting.
I20260620 09:05:22.704067 2736 tablet_bootstrap.cc:654] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.704339 2738 tablet_bootstrap.cc:654] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.704339 2737 tablet_bootstrap.cc:654] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:22.705082 2736 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: No bootstrap required, opened a new log
I20260620 09:05:22.705174 2736 ts_tablet_manager.cc:1397] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:22.705508 2736 raft_consensus.cc:348] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.705596 2736 raft_consensus.cc:374] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.705617 2736 raft_consensus.cc:729] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea3d5dd8ab74fa9b7514825c4fed357, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.705760 2738 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: No bootstrap required, opened a new log
I20260620 09:05:22.705758 2736 consensus_queue.cc:260] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.705832 2738 ts_tablet_manager.cc:1397] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 09:05:22.706022 2737 tablet_bootstrap.cc:492] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: No bootstrap required, opened a new log
I20260620 09:05:22.706022 2736 ts_tablet_manager.cc:1428] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:22.706091 2737 ts_tablet_manager.cc:1397] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:05:22.706190 2738 raft_consensus.cc:348] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.706287 2738 raft_consensus.cc:374] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.706323 2738 raft_consensus.cc:729] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35a63a4022d4e6a8194a9802a688966, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.706446 2738 consensus_queue.cc:260] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.706485 2737 raft_consensus.cc:348] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.706588 2737 raft_consensus.cc:374] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:22.706627 2737 raft_consensus.cc:729] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Initialized, Role: FOLLOWER
I20260620 09:05:22.706714 2738 ts_tablet_manager.cc:1428] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:22.706746 2737 consensus_queue.cc:260] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.707010 2737 ts_tablet_manager.cc:1428] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:22.817138 2716 raft_consensus.cc:482] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:22.817302 2716 raft_consensus.cc:504] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.817739 2716 leader_election.cc:288] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:22.818197 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6cf9066bd7c409da887193e3d8c9dd8" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" is_pre_election: true
I20260620 09:05:22.818204 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6cf9066bd7c409da887193e3d8c9dd8" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966" is_pre_election: true
I20260620 09:05:22.818356 2425 raft_consensus.cc:2454] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:22.818368 2287 raft_consensus.cc:2454] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 0.
I20260620 09:05:22.818709 2501 leader_election.cc:302] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, 8ea3d5dd8ab74fa9b7514825c4fed357; no voters:
I20260620 09:05:22.818882 2716 raft_consensus.cc:2781] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:22.818945 2716 raft_consensus.cc:482] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:22.818974 2716 raft_consensus.cc:3037] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.820003 2716 raft_consensus.cc:504] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.820377 2716 leader_election.cc:288] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Requested vote from peers f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141), 8ea3d5dd8ab74fa9b7514825c4fed357 (127.0.98.2:42919)
I20260620 09:05:22.820712 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6cf9066bd7c409da887193e3d8c9dd8" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357"
I20260620 09:05:22.820688 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6cf9066bd7c409da887193e3d8c9dd8" candidate_uuid: "6ff21e10975743beb479f6ec4e71b6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35a63a4022d4e6a8194a9802a688966"
I20260620 09:05:22.820812 2425 raft_consensus.cc:3037] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.820837 2287 raft_consensus.cc:3037] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:22.821900 2287 raft_consensus.cc:2454] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:22.821905 2425 raft_consensus.cc:2454] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff21e10975743beb479f6ec4e71b6ec in term 1.
I20260620 09:05:22.822182 2503 leader_election.cc:302] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff21e10975743beb479f6ec4e71b6ec, f35a63a4022d4e6a8194a9802a688966; no voters:
I20260620 09:05:22.822348 2716 raft_consensus.cc:2781] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:22.822443 2716 raft_consensus.cc:686] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Becoming Leader. State: Replica: 6ff21e10975743beb479f6ec4e71b6ec, State: Running, Role: LEADER
I20260620 09:05:22.822582 2716 consensus_queue.cc:237] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } }
I20260620 09:05:22.824328 2132 catalog_manager.cc:5495] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6ff21e10975743beb479f6ec4e71b6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff21e10975743beb479f6ec4e71b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43667 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:22.867822 2425 raft_consensus.cc:1261] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:22.867868 2287 raft_consensus.cc:1261] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Refusing update from remote peer 6ff21e10975743beb479f6ec4e71b6ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:22.868221 2716 consensus_queue.cc:1035] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea3d5dd8ab74fa9b7514825c4fed357" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:22.868381 2646 consensus_queue.cc:1035] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35a63a4022d4e6a8194a9802a688966" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:05:22.881346 2707 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 09:05:24.429462 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:24.429638 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:24.432103 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2b0dad344be4dfb9879a00e146a1522: Sending DeleteTablet for 3 replicas of tablet d6cf9066bd7c409da887193e3d8c9dd8
I20260620 09:05:24.432704 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41854
I20260620 09:05:24.432723 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33470
I20260620 09:05:24.432844 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet d6cf9066bd7c409da887193e3d8c9dd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33842
I20260620 09:05:24.433117 2777 tablet_replica.cc:330] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966: stopping tablet replica
I20260620 09:05:24.433259 2777 raft_consensus.cc:2229] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:24.433396 2777 raft_consensus.cc:2258] T d6cf9066bd7c409da887193e3d8c9dd8 P f35a63a4022d4e6a8194a9802a688966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:24.433470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2204
I20260620 09:05:24.435032 2779 tablet_replica.cc:330] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: stopping tablet replica
I20260620 09:05:24.435209 2779 raft_consensus.cc:2229] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:24.435282 2778 tablet_replica.cc:330] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: stopping tablet replica
I20260620 09:05:24.435338 2779 raft_consensus.cc:2258] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:24.435446 2778 raft_consensus.cc:2229] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:24.435662 2778 raft_consensus.cc:2258] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:24.436062 2779 ts_tablet_manager.cc:1905] T d6cf9066bd7c409da887193e3d8c9dd8 P 8ea3d5dd8ab74fa9b7514825c4fed357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:24.436280 2778 ts_tablet_manager.cc:1905] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:05:24.441872 2119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45141 (error 108)
I20260620 09:05:24.442638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2344
I20260620 09:05:24.443667 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:24.442636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.0.1:41868)
0620 09:05:24.443450 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:24.443536 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":102,"threads_started":1}
W20260620 09:05:24.443933 2119 catalog_manager.cc:4584] TS f35a63a4022d4e6a8194a9802a688966 (127.0.98.1:45141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6cf9066bd7c409da887193e3d8c9dd8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45141: connect: Connection refused (error 111)
I20260620 09:05:24.444938 2778 ts_tablet_manager.cc:1918] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 09:05:24.445086 2778 log.cc:1196] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/d6cf9066bd7c409da887193e3d8c9dd8
I20260620 09:05:24.445434 2778 ts_tablet_manager.cc:1939] T d6cf9066bd7c409da887193e3d8c9dd8 P 6ff21e10975743beb479f6ec4e71b6ec: Deleting consensus metadata
I20260620 09:05:24.449007 2117 catalog_manager.cc:4841] TS 6ff21e10975743beb479f6ec4e71b6ec (127.0.98.3:43667): tablet d6cf9066bd7c409da887193e3d8c9dd8 (table test-table [id=77886894a99244e79d04184a1060d455]) successfully deleted
I20260620 09:05:24.452041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2484
I20260620 09:05:24.452078 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:24.451856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42919 (local address 127.0.0.1:33856)
0620 09:05:24.451962 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:24.452009 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:05:24.459407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2100
2026-06-20T09:05:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6540 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:24 dist-test-slave-92t4 krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:24 dist-test-slave-92t4 krb5kdc[2784](info): set up 2 sockets
Jun 20 09:05:24 dist-test-slave-92t4 krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260620 09:05:25.684927 392 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:25 dist-test-slave-92t4 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:25.799926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:25.908818 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:25.909161 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:25.909206 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:25.913045 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:25.913116 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:25.913137 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:25.913159 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:25.913174 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:25.918489 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:25.919486 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:25.920431 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:25.928857 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:25.928844 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:25.928850 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:25.928928 2797 server_base.cc:1034] running on GCE node
I20260620 09:05:25.929733 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:25.930903 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:25.932092 2797 hybrid_clock.cc:648] HybridClock initialized: now 1781946325932078 us; error 34 us; skew 500 ppm
Jun 20 09:05:25 dist-test-slave-92t4 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:25.938189 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:25.939102 2797 webserver.cc:460] Webserver started at http://127.0.98.62:35703/ using document root <none> and password file <none>
I20260620 09:05:25.939674 2797 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:25.939762 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:25.940055 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:25.942312 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "86b99b816ac14daf8bc991a0df28ce36"
format_stamp: "Formatted at 2026-06-20 09:05:25 on dist-test-slave-92t4"
server_key: "77ea8dd4f903e4b0f4660877f39bf351"
server_key_iv: "dccb269277c83d748b37aaaa8d9188e2"
server_key_version: "encryptionkey@0"
I20260620 09:05:25.942870 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "86b99b816ac14daf8bc991a0df28ce36"
format_stamp: "Formatted at 2026-06-20 09:05:25 on dist-test-slave-92t4"
server_key: "77ea8dd4f903e4b0f4660877f39bf351"
server_key_iv: "dccb269277c83d748b37aaaa8d9188e2"
server_key_version: "encryptionkey@0"
I20260620 09:05:25.946597 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:25.948912 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:25.950179 2797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:25.950321 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "86b99b816ac14daf8bc991a0df28ce36"
format_stamp: "Formatted at 2026-06-20 09:05:25 on dist-test-slave-92t4"
server_key: "77ea8dd4f903e4b0f4660877f39bf351"
server_key_iv: "dccb269277c83d748b37aaaa8d9188e2"
server_key_version: "encryptionkey@0"
I20260620 09:05:25.950428 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:25.972205 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:25.975598 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:25.975795 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:25.982635 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43305
I20260620 09:05:25.982649 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43305 every 8 connection(s)
I20260620 09:05:25.983577 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:25.985813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2797
I20260620 09:05:25.985926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:25.986174 392 external_mini_cluster.cc:1428] Setting key 5dc0a7fed329ce9ade4c225dd9b1d97b
I20260620 09:05:25.986436 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:25.993196 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36: Bootstrap starting.
Jun 20 09:05:25 dist-test-slave-92t4 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:25.995507 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:25.996322 2865 log.cc:824] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:25.998237 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36: No bootstrap required, opened a new log
I20260620 09:05:26.000560 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:25.987499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.98.62:43305)
0620 09:05:25.987935 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:25.987951 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:25.987985 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:05:25.988913 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:25.988948 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:25.989379 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:25.989658 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:25.990546 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:25.991515 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:25.992717 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:25.993237 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:25.993395 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:25.996024 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:25.996029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:25.996041 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:25.996080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:25.997859 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:25.998280 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:25.998286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:25.998292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:25.998455 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:25.998735 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:25.998740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:25.998742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:25.999131 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:25.999311 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:25.999586 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 09:05:25.999861 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20260620 09:05:26.001845 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } }
I20260620 09:05:26.002097 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:26.002163 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b99b816ac14daf8bc991a0df28ce36, State: Initialized, Role: FOLLOWER
I20260620 09:05:26.002599 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } }
I20260620 09:05:26.002784 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:26.002854 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:26.002940 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:26.003711 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } }
I20260620 09:05:26.003980 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b99b816ac14daf8bc991a0df28ce36; no voters:
I20260620 09:05:26.004206 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:26.004312 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:26.004508 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [term 1 LEADER]: Becoming Leader. State: Replica: 86b99b816ac14daf8bc991a0df28ce36, State: Running, Role: LEADER
I20260620 09:05:26.004833 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } }
I20260620 09:05:26.005391 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:26.006978 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b99b816ac14daf8bc991a0df28ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } } }
I20260620 09:05:26.007107 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:26.007311 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b99b816ac14daf8bc991a0df28ce36. Latest consensus state: current_term: 1 leader_uuid: "86b99b816ac14daf8bc991a0df28ce36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b99b816ac14daf8bc991a0df28ce36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43305 } } }
I20260620 09:05:26.007424 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:26.007623 2882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:26.010144 2882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:26.014928 2882 catalog_manager.cc:1347] Generated new cluster ID: 6e9fc5e9a3a448028207877c1837e8c4
I20260620 09:05:26.014988 2882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:26.025445 2882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:26.026619 2882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:26.031893 2882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b99b816ac14daf8bc991a0df28ce36: Generated new TSK 0
I20260620 09:05:26.032636 2882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:26.060987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:26.174598 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:26.174860 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:26.174903 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:26.178848 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:26.178898 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:26.178954 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:26.184113 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:26.185108 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:26.186065 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:26.195402 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.195410 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.195386 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:26.195643 2891 server_base.cc:1034] running on GCE node
I20260620 09:05:26.196020 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:26.196578 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:26.197734 2891 hybrid_clock.cc:648] HybridClock initialized: now 1781946326197703 us; error 44 us; skew 500 ppm
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:26.200379 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:26.201314 2891 webserver.cc:460] Webserver started at http://127.0.98.1:34473/ using document root <none> and password file <none>
I20260620 09:05:26.201865 2891 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:26.201952 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:26.202171 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:26.204142 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "3d05f982ac3441da943f9ff780b24c62"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "59dac6452bfdda92bd28aeaba2fadffd"
server_key_iv: "6d13ea3a6f5cc3344d2005aa75fd2887"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.204697 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3d05f982ac3441da943f9ff780b24c62"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "59dac6452bfdda92bd28aeaba2fadffd"
server_key_iv: "6d13ea3a6f5cc3344d2005aa75fd2887"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.208449 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:26.210855 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.211994 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:05:26.212121 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "3d05f982ac3441da943f9ff780b24c62"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "59dac6452bfdda92bd28aeaba2fadffd"
server_key_iv: "6d13ea3a6f5cc3344d2005aa75fd2887"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.212244 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:26.223683 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:26.226872 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:26.227087 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:26.227721 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:26.228509 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:26.228577 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.228658 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:26.228696 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.237104 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35285
I20260620 09:05:26.237156 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35285 every 8 connection(s)
I20260620 09:05:26.237967 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:26.247260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 2891
I20260620 09:05:26.247375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:26.247645 392 external_mini_cluster.cc:1428] Setting key 73f0ec6f01d7f0b89702848188d0f5d7
I20260620 09:05:26.251871 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.239727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45781 (local address 127.0.98.62:43305)
0620 09:05:26.239878 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:26.239883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:26.240731 (+ 848us) server_negotiation.cc:407] Connection header received
0620 09:05:26.242121 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:26.242124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:26.242191 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:26.242289 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:26.243513 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.244057 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.245201 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.245424 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.245502 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.248008 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:26.248013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:26.248015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:26.248042 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:26.249809 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.250267 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.250273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.250277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.250374 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.250736 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.250742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.250745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.250956 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:26.251036 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:26.251642 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 09:05:26.251739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:05:26.253199 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.240116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43305 (local address 127.0.98.1:45781)
0620 09:05:26.240577 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:26.240621 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:26.241922 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:26.242539 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:26.242546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:26.242923 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:26.243382 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.243395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.244171 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.244173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.245102 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.245116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.245549 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.245552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.245704 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.246238 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:26.246253 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:26.247845 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:26.249915 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.249919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.249930 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.250174 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.250462 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.250464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.250466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.250587 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.251055 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:26.251065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:26.251396 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 09:05:26.251608 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 09:05:26.252075 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260620 09:05:26.254478 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43305
I20260620 09:05:26.254776 3020 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:26.255378 3020 heartbeater.cc:511] Master 127.0.98.62:43305 requested a full tablet report, sending...
I20260620 09:05:26.257121 2829 ts_manager.cc:194] Registered new tserver with Master: 3d05f982ac3441da943f9ff780b24c62 (127.0.98.1:35285)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:26.258455 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:26.273392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:26.383370 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:26.383672 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:26.383718 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:26.387616 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:26.387668 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:26.387723 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:26.392724 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:26.393668 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:26.394579 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:26.403748 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.403785 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.403806 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:26.404069 3025 server_base.cc:1034] running on GCE node
I20260620 09:05:26.404502 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:26.405194 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:26.406361 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781946326406326 us; error 48 us; skew 500 ppm
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:26.409083 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:26.409929 3025 webserver.cc:460] Webserver started at http://127.0.98.2:45139/ using document root <none> and password file <none>
I20260620 09:05:26.410487 3025 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:26.410575 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:26.410813 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:26.412846 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4702459ca0b4bcb881e620995511e8a"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "261a56d6c75efd0d4b32bbae7417f060"
server_key_iv: "84c82a74e52e5f49bac4eb9d02d803fb"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.413440 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4702459ca0b4bcb881e620995511e8a"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "261a56d6c75efd0d4b32bbae7417f060"
server_key_iv: "84c82a74e52e5f49bac4eb9d02d803fb"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.417210 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:26.419550 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.420735 3025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:26.420868 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "d4702459ca0b4bcb881e620995511e8a"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "261a56d6c75efd0d4b32bbae7417f060"
server_key_iv: "84c82a74e52e5f49bac4eb9d02d803fb"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.420975 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:26.441985 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:26.444937 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:26.445158 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:26.445747 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:26.446563 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:26.446636 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.446722 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:26.446765 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.455729 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36239
I20260620 09:05:26.455756 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36239 every 8 connection(s)
I20260620 09:05:26.456598 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:26.459136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3025
I20260620 09:05:26.459254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:26.459533 392 external_mini_cluster.cc:1428] Setting key 0c307cfced74d727611891845e3dda4a
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:26.470439 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.458300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34307 (local address 127.0.98.62:43305)
0620 09:05:26.458443 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:26.458447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:26.459323 (+ 876us) server_negotiation.cc:407] Connection header received
0620 09:05:26.460728 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:26.460732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:26.460780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:26.460863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:26.462037 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.462565 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.463703 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.463924 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.464032 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.466814 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:26.466817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:26.466819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:26.466841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:26.468303 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.468762 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.468765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.468766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.468835 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.469192 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.469197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.469199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.469369 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:26.469448 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:26.470097 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 09:05:26.470243 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:05:26.471417 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.458706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43305 (local address 127.0.98.2:34307)
0620 09:05:26.459164 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:26.459230 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:26.460532 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:26.461073 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:26.461082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:26.461440 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:26.461907 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.461919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.462691 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.462694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.463580 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.463589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.464067 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.464071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.464235 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.464821 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:26.464838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:26.466656 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:26.468406 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.468414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.468430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.468681 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.468909 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.468911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.468912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.469108 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.469478 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:26.469482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:26.469769 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 09:05:26.469997 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 09:05:26.470343 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":140,"threads_started":1}
I20260620 09:05:26.472852 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43305
I20260620 09:05:26.473135 3154 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:26.473626 3154 heartbeater.cc:511] Master 127.0.98.62:43305 requested a full tablet report, sending...
I20260620 09:05:26.474638 2829 ts_manager.cc:194] Registered new tserver with Master: d4702459ca0b4bcb881e620995511e8a (127.0.98.2:36239)
I20260620 09:05:26.475472 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:26.485595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:26.598167 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:26.598417 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:26.598459 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:26.602562 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:26.602618 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:26.602675 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:26.608167 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:26.609208 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:26.610206 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:26.619575 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.619617 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:26.619738 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:26.619848 3159 server_base.cc:1034] running on GCE node
I20260620 09:05:26.620348 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:26.620949 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:26.622123 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781946326622105 us; error 29 us; skew 500 ppm
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:26.624732 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:26.625612 3159 webserver.cc:460] Webserver started at http://127.0.98.3:37277/ using document root <none> and password file <none>
I20260620 09:05:26.626135 3159 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:26.626190 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:26.626344 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:26.628301 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c65fa797f8247c29b8ff64789431a00"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "7ef37255fcb9068a9b55ff6b2dfe0172"
server_key_iv: "e0e8f1b1b058b2e52a6a0f6a5043a4e3"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.628790 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c65fa797f8247c29b8ff64789431a00"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "7ef37255fcb9068a9b55ff6b2dfe0172"
server_key_iv: "e0e8f1b1b058b2e52a6a0f6a5043a4e3"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.632269 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:26.634567 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.635715 3159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:26.635812 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "6c65fa797f8247c29b8ff64789431a00"
format_stamp: "Formatted at 2026-06-20 09:05:26 on dist-test-slave-92t4"
server_key: "7ef37255fcb9068a9b55ff6b2dfe0172"
server_key_iv: "e0e8f1b1b058b2e52a6a0f6a5043a4e3"
server_key_version: "encryptionkey@0"
I20260620 09:05:26.635890 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:26.679127 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:26.681977 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:26.682140 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:26.682719 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:26.683501 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:26.683544 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.683607 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:26.683641 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:26.692273 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41041
I20260620 09:05:26.692323 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41041 every 8 connection(s)
I20260620 09:05:26.693212 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:26.702343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3159
I20260620 09:05:26.702458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:26.702777 392 external_mini_cluster.cc:1428] Setting key 54d9587fd6932ca0b17fd54107d42b58
I20260620 09:05:26.706265 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.694967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35089 (local address 127.0.98.62:43305)
0620 09:05:26.695097 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:26.695100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:26.695868 (+ 768us) server_negotiation.cc:407] Connection header received
0620 09:05:26.697195 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:26.697198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:26.697245 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:26.697317 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:26.698430 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.698942 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.700007 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.700213 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.700284 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.702682 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:26.702686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:26.702689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:26.702717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:26.704134 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.704682 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.704687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.704689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.704774 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:26.705141 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:26.705144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:26.705153 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:26.705331 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:26.705432 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:26.705998 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 09:05:26.706137 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:05:26.707181 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:26.695293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43305 (local address 127.0.98.3:35089)
0620 09:05:26.695708 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:26.695749 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:26.696958 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:26.697473 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:26.697479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:26.697811 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:26.698308 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.698322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.699039 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.699041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.699910 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:26.699918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:26.700326 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:26.700328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:26.700480 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:26.700976 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:26.700990 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:26.702479 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:26.704232 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.704238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.704251 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.704581 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.704873 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:26.704877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:26.704882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:26.705007 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:26.705431 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:26.705437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:26.705750 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 09:05:26.705916 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 09:05:26.706213 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260620 09:05:26.708141 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43305
I20260620 09:05:26.708379 3288 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:26.708840 3288 heartbeater.cc:511] Master 127.0.98.62:43305 requested a full tablet report, sending...
I20260620 09:05:26.709815 2829 ts_manager.cc:194] Registered new tserver with Master: 6c65fa797f8247c29b8ff64789431a00 (127.0.98.3:41041)
I20260620 09:05:26.710374 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35089
I20260620 09:05:26.716316 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:05:26.728717 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:26.727272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.98.62:43305)
0620 09:05:26.727422 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:26.727428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:26.727575 (+ 147us) server_negotiation.cc:407] Connection header received
0620 09:05:26.727691 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:26.727694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:26.727737 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:26.727804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:26.728559 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40098: BlockingRecv error: recv got EOF from 127.0.0.1:40098 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:05:26.729734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2891
I20260620 09:05:26.735348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3025
I20260620 09:05:26.740576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3159
I20260620 09:05:26.745689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 2797
2026-06-20T09:05:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2283 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[3300](info): set up 2 sockets
Jun 20 09:05:26 dist-test-slave-92t4 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260620 09:05:27.773062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:27 dist-test-slave-92t4 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:27.891387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:27.999986 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:28.000296 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:28.000367 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:28.004241 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:28.004344 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:28.004381 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:28.004417 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:28.004460 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:28.009636 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:28.010716 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:28.011776 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:28.019972 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:28.019966 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:28.020177 3313 server_base.cc:1034] running on GCE node
W20260620 09:05:28.019999 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:28.020865 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:28.021981 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:28.023131 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781946328023118 us; error 26 us; skew 500 ppm
Jun 20 09:05:28 dist-test-slave-92t4 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:28.025745 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:28.026656 3313 webserver.cc:460] Webserver started at http://127.0.98.62:38519/ using document root <none> and password file <none>
I20260620 09:05:28.027200 3313 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:28.027258 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:28.027500 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:28.029575 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "31d3b095f31c46c3b73a72f4fc436574"
format_stamp: "Formatted at 2026-06-20 09:05:28 on dist-test-slave-92t4"
server_key: "d883a655f83b29f5955c1738ad404507"
server_key_iv: "271fbafc55eeb9bb30eff333526266d5"
server_key_version: "encryptionkey@0"
I20260620 09:05:28.030195 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "31d3b095f31c46c3b73a72f4fc436574"
format_stamp: "Formatted at 2026-06-20 09:05:28 on dist-test-slave-92t4"
server_key: "d883a655f83b29f5955c1738ad404507"
server_key_iv: "271fbafc55eeb9bb30eff333526266d5"
server_key_version: "encryptionkey@0"
I20260620 09:05:28.033845 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:28.036108 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:28.037451 3313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:28.037546 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "31d3b095f31c46c3b73a72f4fc436574"
format_stamp: "Formatted at 2026-06-20 09:05:28 on dist-test-slave-92t4"
server_key: "d883a655f83b29f5955c1738ad404507"
server_key_iv: "271fbafc55eeb9bb30eff333526266d5"
server_key_version: "encryptionkey@0"
I20260620 09:05:28.037638 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:28.048594 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:28.051859 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:28.052019 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:28.058774 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45269
I20260620 09:05:28.059059 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45269 every 8 connection(s)
I20260620 09:05:28.059770 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:28.062592 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:28.067261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3313
I20260620 09:05:28.067387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:28.067688 392 external_mini_cluster.cc:1428] Setting key f2a98c7fd21103dfbf763d12876a6f2d
I20260620 09:05:28.068444 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574: Bootstrap starting.
I20260620 09:05:28.071246 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:28.072094 3381 log.cc:824] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:28.074093 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574: No bootstrap required, opened a new log
Jun 20 09:05:28 dist-test-slave-92t4 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:28.078341 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260620 09:05:28.078660 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:28.078722 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d3b095f31c46c3b73a72f4fc436574, State: Initialized, Role: FOLLOWER
I20260620 09:05:28.079211 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260620 09:05:28.079376 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:28.079447 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:28.079564 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:28.080747 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260620 09:05:28.081130 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d3b095f31c46c3b73a72f4fc436574; no voters:
I20260620 09:05:28.081492 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:28.081827 3387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:28.082019 3387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [term 1 LEADER]: Becoming Leader. State: Replica: 31d3b095f31c46c3b73a72f4fc436574, State: Running, Role: LEADER
I20260620 09:05:28.082377 3387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260620 09:05:28.082630 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:28.068964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.0.98.62:45269)
0620 09:05:28.069442 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:28.069461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:28.069502 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:05:28.070353 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:28.070375 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:28.070782 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:28.071142 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:28.071873 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:28.073232 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:28.074059 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:28.074722 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:28.074877 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:28.077516 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:28.077523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:28.077541 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:28.077571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:28.079452 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:28.079959 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:28.079965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:28.079971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:28.080140 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:28.080384 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:28.080389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:28.080392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:28.080911 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:28.081141 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:28.081394 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 09:05:28.081693 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":114,"threads_started":1}
I20260620 09:05:28.082713 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:28.083557 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d3b095f31c46c3b73a72f4fc436574. Latest consensus state: current_term: 1 leader_uuid: "31d3b095f31c46c3b73a72f4fc436574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260620 09:05:28.083765 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:28.084004 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d3b095f31c46c3b73a72f4fc436574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3b095f31c46c3b73a72f4fc436574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260620 09:05:28.084090 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:28.084257 3395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:28.085813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:05:28.087745 3395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:28.093918 3395 catalog_manager.cc:1347] Generated new cluster ID: 3b95b46054cc408e8410850fa282bfa5
I20260620 09:05:28.094022 3395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:28.107945 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:28.102273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.0.98.62:45269)
0620 09:05:28.102388 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:28.102394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:28.102591 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:05:28.102744 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:28.102747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:28.102797 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:28.102868 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:28.103473 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:28.104260 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:28.105494 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:28.105868 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:28.105978 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:28.106834 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:28.106837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:28.106839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:28.107403 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:28.107432 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:28.107448 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:28.107556 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:28.107741 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:05:28.107801 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:05:28.107990 3395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:28.109162 3395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:28.119633 3395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d3b095f31c46c3b73a72f4fc436574: Generated new TSK 0
I20260620 09:05:28.120294 3395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:05:28.126168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3313
2026-06-20T09:05:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1378 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:28 dist-test-slave-92t4 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:28 dist-test-slave-92t4 krb5kdc[3414](info): set up 2 sockets
Jun 20 09:05:28 dist-test-slave-92t4 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260620 09:05:29.157012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:29.277410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:29.391081 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:29.391361 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:29.391402 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:29.395371 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:29.395428 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:29.395448 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:29.395464 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:29.395479 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:29.400643 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:29.401690 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:29.402657 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:29.411065 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.411020 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.411042 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:29.411448 3427 server_base.cc:1034] running on GCE node
I20260620 09:05:29.411976 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:29.413107 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:29.414296 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781946329414278 us; error 35 us; skew 500 ppm
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:29.416982 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:29.417878 3427 webserver.cc:460] Webserver started at http://127.0.98.62:36433/ using document root <none> and password file <none>
I20260620 09:05:29.418380 3427 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:29.418435 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:29.418594 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:29.420526 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "bf335112f48ffa89c6c7b8915bdb898d"
server_key_iv: "2d27627c8051beca724b6ac275162ce1"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.421096 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "bf335112f48ffa89c6c7b8915bdb898d"
server_key_iv: "2d27627c8051beca724b6ac275162ce1"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.424685 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:05:29.427078 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.428328 3427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:29.428454 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "bf335112f48ffa89c6c7b8915bdb898d"
server_key_iv: "2d27627c8051beca724b6ac275162ce1"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.428555 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:29.468501 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:29.471973 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:29.472182 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:29.479310 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34699
I20260620 09:05:29.479621 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34699 every 8 connection(s)
I20260620 09:05:29.480435 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:29.483809 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:29.484706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3427
I20260620 09:05:29.484848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:29.485186 392 external_mini_cluster.cc:1428] Setting key 95197b38dea5d0a3eced92bb71f1a3a7
I20260620 09:05:29.490686 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae: Bootstrap starting.
I20260620 09:05:29.493227 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:29.493940 3495 log.cc:824] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:29.496202 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae: No bootstrap required, opened a new log
I20260620 09:05:29.500660 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:29.486571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.62:34699)
0620 09:05:29.486988 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:29.487008 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:29.487060 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:05:29.487893 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:29.487914 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:29.488325 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:29.488689 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:29.489410 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.490835 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.491664 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.492334 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.492481 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:29.495441 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:29.495447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:29.495464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:29.495493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:29.497411 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.497840 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.497845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.497851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.498001 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.498261 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.498266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.498268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.498835 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:29.499055 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:29.499373 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 09:05:29.499684 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260620 09:05:29.500841 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } }
I20260620 09:05:29.501163 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:29.501222 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf19b0c540df4e0b8a8fe6325af9f5ae, State: Initialized, Role: FOLLOWER
I20260620 09:05:29.501698 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } }
I20260620 09:05:29.501868 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:29.501933 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:29.502020 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:29.503077 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } }
I20260620 09:05:29.503379 3495 leader_election.cc:302] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf19b0c540df4e0b8a8fe6325af9f5ae; no voters:
I20260620 09:05:29.503654 3495 leader_election.cc:288] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:29.503780 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:29.504009 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [term 1 LEADER]: Becoming Leader. State: Replica: bf19b0c540df4e0b8a8fe6325af9f5ae, State: Running, Role: LEADER
I20260620 09:05:29.504341 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } }
I20260620 09:05:29.504825 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:29.506131 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf19b0c540df4e0b8a8fe6325af9f5ae. Latest consensus state: current_term: 1 leader_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } } }
I20260620 09:05:29.506151 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf19b0c540df4e0b8a8fe6325af9f5ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34699 } } }
I20260620 09:05:29.506269 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:29.506271 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:29.506597 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:29.509825 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:29.514971 3509 catalog_manager.cc:1347] Generated new cluster ID: e52e7debd03c4fe5abc08321fe348f79
I20260620 09:05:29.515029 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:29.534521 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:29.535740 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:29.545444 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf19b0c540df4e0b8a8fe6325af9f5ae: Generated new TSK 0
I20260620 09:05:29.546247 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:29.575981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:29.685200 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:29.685477 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:29.685552 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:29.689555 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:29.689628 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:29.689733 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:29.694875 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:29.695876 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:29.696861 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:29.705967 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.705962 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.705976 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:29.706322 3521 server_base.cc:1034] running on GCE node
I20260620 09:05:29.706727 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:29.707350 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:29.708528 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781946329708511 us; error 32 us; skew 500 ppm
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:29.711162 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:29.711961 3521 webserver.cc:460] Webserver started at http://127.0.98.1:40567/ using document root <none> and password file <none>
I20260620 09:05:29.712445 3521 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:29.712498 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:29.712662 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:29.714649 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b21278a8c594d79837cd768c39bd9dc"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "2483fae2bc7d20bd9157eda818b67699"
server_key_iv: "b3b3a20036669e9b87a69198dd0d4849"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.715143 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b21278a8c594d79837cd768c39bd9dc"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "2483fae2bc7d20bd9157eda818b67699"
server_key_iv: "b3b3a20036669e9b87a69198dd0d4849"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.718647 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 09:05:29.720789 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.721995 3521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:29.722083 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "5b21278a8c594d79837cd768c39bd9dc"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "2483fae2bc7d20bd9157eda818b67699"
server_key_iv: "b3b3a20036669e9b87a69198dd0d4849"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.722159 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:29.732887 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:29.736210 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:29.736357 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:29.736883 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:29.737679 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:29.737722 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.737759 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:29.737782 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.746592 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37273
I20260620 09:05:29.746624 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37273 every 8 connection(s)
I20260620 09:05:29.747424 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:29.752202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3521
I20260620 09:05:29.752292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:29.752563 392 external_mini_cluster.cc:1428] Setting key 0ea9d0c896570a97bb7dc782329c5cb3
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:29.762254 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:29.749170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55289 (local address 127.0.98.62:34699)
0620 09:05:29.749326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:29.749331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:29.750261 (+ 930us) server_negotiation.cc:407] Connection header received
0620 09:05:29.751746 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:29.751748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:29.751791 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:29.751864 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:29.753194 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.753922 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.755642 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.756006 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.756141 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:29.758615 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:29.758619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:29.758622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:29.758655 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:29.760047 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.760540 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.760544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.760546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.760625 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.760981 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.760984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.760985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.761238 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:29.761349 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:29.762038 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 09:05:29.762140 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:29.763226 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:29.749619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34699 (local address 127.0.98.1:55289)
0620 09:05:29.750100 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:29.750151 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:29.751563 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:29.752035 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:29.752043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:29.752459 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:29.753011 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:29.753023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.754052 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.754056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:29.755507 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:29.755521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.756164 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.756168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:29.756362 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:29.756996 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:29.757013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:29.758461 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:29.760181 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:29.760186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:29.760199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:29.760440 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:29.760729 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:29.760733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:29.760735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:29.760870 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:29.761369 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:29.761375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:29.761694 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 09:05:29.761964 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 09:05:29.762302 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260620 09:05:29.764412 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34699
I20260620 09:05:29.764699 3650 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:05:29.765242 3650 heartbeater.cc:511] Master 127.0.98.62:34699 requested a full tablet report, sending...
I20260620 09:05:29.766681 3459 ts_manager.cc:194] Registered new tserver with Master: 5b21278a8c594d79837cd768c39bd9dc (127.0.98.1:37273)
I20260620 09:05:29.767781 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:29.779311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:29.894392 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:29.894636 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:29.894690 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:29.898783 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:29.898835 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:29.898890 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:29.904320 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:29.905412 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:29.906448 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:29.916016 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.915946 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:29.915944 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:29.916440 3655 server_base.cc:1034] running on GCE node
I20260620 09:05:29.916929 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:29.917572 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:29.918735 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781946329918723 us; error 24 us; skew 500 ppm
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:29.921332 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:29.922107 3655 webserver.cc:460] Webserver started at http://127.0.98.2:39739/ using document root <none> and password file <none>
I20260620 09:05:29.922662 3655 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:29.922751 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:29.922972 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:29.925029 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "9a48efb7616b457884dfa21da62f8eb3"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "572567c6f12aca71f5882baefd14f303"
server_key_iv: "f1c4ad5ff0250d4dbdd9c1a5dd414e53"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.925714 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9a48efb7616b457884dfa21da62f8eb3"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "572567c6f12aca71f5882baefd14f303"
server_key_iv: "f1c4ad5ff0250d4dbdd9c1a5dd414e53"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.929380 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:05:29.931620 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.932890 3655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:29.933075 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "9a48efb7616b457884dfa21da62f8eb3"
format_stamp: "Formatted at 2026-06-20 09:05:29 on dist-test-slave-92t4"
server_key: "572567c6f12aca71f5882baefd14f303"
server_key_iv: "f1c4ad5ff0250d4dbdd9c1a5dd414e53"
server_key_version: "encryptionkey@0"
I20260620 09:05:29.933209 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:29.951979 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:29.954897 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:29.955101 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:29.955682 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:29.956466 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:29.956537 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.956602 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:29.956657 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:29.964784 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36381
I20260620 09:05:29.964854 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36381 every 8 connection(s)
I20260620 09:05:29.965708 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:29.975308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3655
I20260620 09:05:29.975405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
Jun 20 09:05:29 dist-test-slave-92t4 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:29.975769 392 external_mini_cluster.cc:1428] Setting key 7d0f4decdb00e05bdfa20184d73ed929
I20260620 09:05:29.979816 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:29.967521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56819 (local address 127.0.98.62:34699)
0620 09:05:29.967659 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:29.967662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:29.968434 (+ 772us) server_negotiation.cc:407] Connection header received
0620 09:05:29.970288 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:29.970291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:29.970345 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:29.970454 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:29.971578 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.972110 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.973324 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.973546 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.973657 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:29.976122 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:29.976128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:29.976133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:29.976162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:29.977576 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.978202 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.978204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.978205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.978278 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:29.978653 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:29.978655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:29.978656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:29.978922 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:29.979028 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:29.979473 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 09:05:29.979600 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:05:29.980501 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:29.967875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34699 (local address 127.0.98.2:56819)
0620 09:05:29.968289 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:29.968334 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:29.970081 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:29.970626 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:29.970633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:29.970964 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:29.971434 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:29.971446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.972247 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.972250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:29.973192 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:29.973201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:29.973694 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:29.973698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:29.973860 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:29.974391 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:29.974413 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:29.975970 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:29.977717 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:29.977726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:29.977743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:29.978104 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:29.978364 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:29.978370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:29.978375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:29.978565 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:29.979028 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:29.979033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:29.979247 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 09:05:29.979398 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 09:05:29.979639 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260620 09:05:29.981861 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34699
I20260620 09:05:29.982096 3784 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:29.982535 3784 heartbeater.cc:511] Master 127.0.98.62:34699 requested a full tablet report, sending...
I20260620 09:05:29.983533 3459 ts_manager.cc:194] Registered new tserver with Master: 9a48efb7616b457884dfa21da62f8eb3 (127.0.98.2:36381)
I20260620 09:05:29.984089 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56819
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 09:05:30.257174 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.280s user 0.002s sys 0.002s
I20260620 09:05:30.257679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:30.366825 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:30.367087 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:30.367137 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:30.371562 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:30.371622 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:30.371680 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:30.376832 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:30.377880 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:30.378823 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:30.388340 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:30.388340 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:30.388562 3789 server_base.cc:1034] running on GCE node
W20260620 09:05:30.388504 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:30.388993 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:30.389604 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:30.390779 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781946330390746 us; error 40 us; skew 500 ppm
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:30.393925 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:30.394709 3789 webserver.cc:460] Webserver started at http://127.0.98.3:34775/ using document root <none> and password file <none>
I20260620 09:05:30.395201 3789 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:30.395257 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:30.395412 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:30.397401 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "bcc16eb8a5de4fedb89ee0a1c4e2fca6"
format_stamp: "Formatted at 2026-06-20 09:05:30 on dist-test-slave-92t4"
server_key: "f6bada382763372080385addf5b41b30"
server_key_iv: "5d135b0cf0af8b53f5577aa2288d02e8"
server_key_version: "encryptionkey@0"
I20260620 09:05:30.397927 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bcc16eb8a5de4fedb89ee0a1c4e2fca6"
format_stamp: "Formatted at 2026-06-20 09:05:30 on dist-test-slave-92t4"
server_key: "f6bada382763372080385addf5b41b30"
server_key_iv: "5d135b0cf0af8b53f5577aa2288d02e8"
server_key_version: "encryptionkey@0"
I20260620 09:05:30.401342 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:05:30.403577 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:30.404678 3789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:30.404764 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "bcc16eb8a5de4fedb89ee0a1c4e2fca6"
format_stamp: "Formatted at 2026-06-20 09:05:30 on dist-test-slave-92t4"
server_key: "f6bada382763372080385addf5b41b30"
server_key_iv: "5d135b0cf0af8b53f5577aa2288d02e8"
server_key_version: "encryptionkey@0"
I20260620 09:05:30.404840 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:30.419163 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:30.422391 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:30.422567 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:30.423061 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:30.423810 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:30.423853 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:30.423890 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:30.423911 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:30.432868 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39315
I20260620 09:05:30.432890 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39315 every 8 connection(s)
I20260620 09:05:30.433764 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:30.443719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3789
I20260620 09:05:30.443858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:30.444124 392 external_mini_cluster.cc:1428] Setting key dc90f0120d491d0aaa1270f7df9e311a
I20260620 09:05:30.447101 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:30.435494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40019 (local address 127.0.98.62:34699)
0620 09:05:30.435644 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:30.435647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:30.436440 (+ 793us) server_negotiation.cc:407] Connection header received
0620 09:05:30.438054 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:30.438057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:30.438107 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:30.438182 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:30.439300 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.439856 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.440909 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.441198 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.441274 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:30.443541 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:30.443544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:30.443546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:30.443568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:30.445253 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:30.445808 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:30.445810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:30.445811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:30.445881 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:30.446216 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:30.446219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:30.446220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:30.446411 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:30.446487 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:30.446902 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 09:05:30.446992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 09:05:30.447886 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:30.435843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34699 (local address 127.0.98.3:40019)
0620 09:05:30.436257 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:30.436298 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:30.437845 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:30.438349 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:30.438356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:30.438699 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:30.439142 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:30.439154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.439955 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.439957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:30.440809 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:30.440817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.441327 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.441330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:30.441460 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:30.441975 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:30.441990 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:30.443288 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:30.445352 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:30.445359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:30.445373 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:30.445714 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:30.445963 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:30.445966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:30.445969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:30.446144 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:30.446485 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:30.446489 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:30.446701 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 09:05:30.446855 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 09:05:30.447076 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260620 09:05:30.448872 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34699
I20260620 09:05:30.449133 3918 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:30.449621 3918 heartbeater.cc:511] Master 127.0.98.62:34699 requested a full tablet report, sending...
I20260620 09:05:30.450587 3459 ts_manager.cc:194] Registered new tserver with Master: bcc16eb8a5de4fedb89ee0a1c4e2fca6 (127.0.98.3:39315)
I20260620 09:05:30.451364 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40019
I20260620 09:05:30.457763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946330, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946330, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:30.481819 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:30.473734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.0.98.62:34699)
0620 09:05:30.473890 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:30.473894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:30.474027 (+ 133us) server_negotiation.cc:407] Connection header received
0620 09:05:30.474202 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:30.474204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:30.474245 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:30.474327 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:30.475064 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.475571 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.476369 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:30.476560 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:30.476636 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:30.478880 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:30.478883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:30.478885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:30.478905 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:30.480248 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:30.480676 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:30.480678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:30.480680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:30.480745 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:30.481016 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:30.481019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:30.481021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:30.481261 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:30.481363 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:30.481592 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 09:05:30.481678 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 09:05:30.482168 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42098
I20260620 09:05:30.482949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3521
I20260620 09:05:30.489812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3655
I20260620 09:05:30.496024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3789
I20260620 09:05:30.501302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3427
2026-06-20T09:05:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2376 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3930](info): set up 2 sockets
Jun 20 09:05:30 dist-test-slave-92t4 krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260620 09:05:31.521126 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:31 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:31.639328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:31.747031 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:31.747274 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:31.747311 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:31.751325 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:31.751384 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:31.751402 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:31.751418 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:31.751434 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:31.756732 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:31.757819 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:31.758844 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:31.767251 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:31.767306 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:31.767222 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:31.767555 3943 server_base.cc:1034] running on GCE node
I20260620 09:05:31.768134 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:31.769258 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:31.770430 3943 hybrid_clock.cc:648] HybridClock initialized: now 1781946331770417 us; error 25 us; skew 500 ppm
Jun 20 09:05:31 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:31.776031 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:31.776894 3943 webserver.cc:460] Webserver started at http://127.0.98.62:41037/ using document root <none> and password file <none>
I20260620 09:05:31.777487 3943 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:31.777606 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:31.777832 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:31.779866 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "06bfa346a7bc40d8bd8660e15f590026"
format_stamp: "Formatted at 2026-06-20 09:05:31 on dist-test-slave-92t4"
server_key: "a1c122dbc9928ac8c2e57c3396ded0ef"
server_key_iv: "ba5f2b905b944007bb2d85aa9d54e063"
server_key_version: "encryptionkey@0"
I20260620 09:05:31.780434 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "06bfa346a7bc40d8bd8660e15f590026"
format_stamp: "Formatted at 2026-06-20 09:05:31 on dist-test-slave-92t4"
server_key: "a1c122dbc9928ac8c2e57c3396ded0ef"
server_key_iv: "ba5f2b905b944007bb2d85aa9d54e063"
server_key_version: "encryptionkey@0"
I20260620 09:05:31.784070 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:31.786388 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:31.787606 3943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:31.787740 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "06bfa346a7bc40d8bd8660e15f590026"
format_stamp: "Formatted at 2026-06-20 09:05:31 on dist-test-slave-92t4"
server_key: "a1c122dbc9928ac8c2e57c3396ded0ef"
server_key_iv: "ba5f2b905b944007bb2d85aa9d54e063"
server_key_version: "encryptionkey@0"
I20260620 09:05:31.787848 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:31.798404 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:31.801081 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:31.801271 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:31.807920 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34489
I20260620 09:05:31.807930 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34489 every 8 connection(s)
I20260620 09:05:31.808820 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:31.811470 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:31.815572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 3943
I20260620 09:05:31.815694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:31.815944 392 external_mini_cluster.cc:1428] Setting key 8beb08f1e3b8a0e2e8cf5619bcf4fac5
I20260620 09:05:31.817134 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: Bootstrap starting.
I20260620 09:05:31.819711 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:31.820569 4011 log.cc:824] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:31.822598 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: No bootstrap required, opened a new log
Jun 20 09:05:31 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:31.826562 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 09:05:31.826756 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:31.826879 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bfa346a7bc40d8bd8660e15f590026, State: Initialized, Role: FOLLOWER
I20260620 09:05:31.827384 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 09:05:31.827587 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:31.827661 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:31.827762 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:31.828835 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 09:05:31.829210 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06bfa346a7bc40d8bd8660e15f590026; no voters:
I20260620 09:05:31.829571 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:31.829670 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:31.829962 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [term 1 LEADER]: Becoming Leader. State: Replica: 06bfa346a7bc40d8bd8660e15f590026, State: Running, Role: LEADER
I20260620 09:05:31.830322 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 09:05:31.830878 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:31.831022 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:31.817344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.62:34489)
0620 09:05:31.817735 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:31.817752 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:31.817792 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:05:31.818628 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:31.818651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:31.819046 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:31.819387 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:31.820049 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:31.821417 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:31.822242 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:31.822894 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:31.823418 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:31.825801 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:31.825805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:31.825818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:31.825843 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:31.827846 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:31.828247 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:31.828253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:31.828258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:31.828393 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:31.828639 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:31.828646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:31.828649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:31.829166 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:31.829352 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:31.829651 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 09:05:31.830085 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260620 09:05:31.831825 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06bfa346a7bc40d8bd8660e15f590026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260620 09:05:31.832016 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:31.832285 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06bfa346a7bc40d8bd8660e15f590026. Latest consensus state: current_term: 1 leader_uuid: "06bfa346a7bc40d8bd8660e15f590026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06bfa346a7bc40d8bd8660e15f590026" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260620 09:05:31.832413 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026 [sys.catalog]: This master's current role is: LEADER
W20260620 09:05:31.835582 4031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:31.835644 4031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:31.835721 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:31.836393 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:31.841260 4025 catalog_manager.cc:1347] Generated new cluster ID: 2b8fe0e18daf4ba7b417a2da006b9776
I20260620 09:05:31.841318 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:31.852373 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:31.853233 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:31.864725 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06bfa346a7bc40d8bd8660e15f590026: Generated new TSK 0
I20260620 09:05:31.865387 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:31.898290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:32.008620 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:32.008869 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:32.008912 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:32.012892 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:32.012944 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:32.013000 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:32.018249 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:32.019282 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:32.020316 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:32.029510 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.029544 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.029578 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:32.029767 4037 server_base.cc:1034] running on GCE node
I20260620 09:05:32.030166 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:32.030862 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:32.032029 4037 hybrid_clock.cc:648] HybridClock initialized: now 1781946332032015 us; error 27 us; skew 500 ppm
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:32.034819 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:32.035635 4037 webserver.cc:460] Webserver started at http://127.0.98.1:38571/ using document root <none> and password file <none>
I20260620 09:05:32.036186 4037 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:32.036275 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:32.036512 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:32.038483 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "ecd2565e59304cd1a21318b035529384"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "4ec0b889e26ad8b247cc02db65ef6f67"
server_key_iv: "f455773d96d68ad83bea10bd0283519c"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.039033 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ecd2565e59304cd1a21318b035529384"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "4ec0b889e26ad8b247cc02db65ef6f67"
server_key_iv: "f455773d96d68ad83bea10bd0283519c"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.042621 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:05:32.044931 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.046119 4037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:32.046248 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "ecd2565e59304cd1a21318b035529384"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "4ec0b889e26ad8b247cc02db65ef6f67"
server_key_iv: "f455773d96d68ad83bea10bd0283519c"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.046353 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:32.077345 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:32.080184 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:32.080391 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:32.080981 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:32.081845 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:32.081916 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.081998 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:32.082048 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.090880 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45963
I20260620 09:05:32.090911 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45963 every 8 connection(s)
I20260620 09:05:32.091781 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:32.094501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4037
I20260620 09:05:32.094660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:32.094966 392 external_mini_cluster.cc:1428] Setting key 64ea92a3c840f2986de628f14fc5454d
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:32.106138 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.093611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37707 (local address 127.0.98.62:34489)
0620 09:05:32.093761 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:32.093764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:32.094623 (+ 859us) server_negotiation.cc:407] Connection header received
0620 09:05:32.096339 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:32.096342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:32.096391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:32.096488 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:32.098075 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.098577 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.099718 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.099954 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.100065 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.102494 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:32.102498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:32.102501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:32.102536 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:32.104008 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.104499 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.104503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.104505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.104619 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.104948 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.104953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.104955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.105188 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:32.105299 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:32.105855 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 09:05:32.105973 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:05:32.107105 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.093970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.1:37707)
0620 09:05:32.094455 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:32.094513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:32.096109 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:32.096670 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:32.096678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:32.097240 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:32.097930 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.097950 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.098687 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.098690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.099596 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.099604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.100101 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.100106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.100278 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.100866 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:32.100881 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:32.102339 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:32.104131 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.104137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.104150 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.104397 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.104716 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.104721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.104723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.104855 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.105296 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:32.105302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:32.105616 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 09:05:32.105818 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 09:05:32.106128 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260620 09:05:32.108278 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 09:05:32.108501 4166 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:32.108973 4166 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 09:05:32.110602 3975 ts_manager.cc:194] Registered new tserver with Master: ecd2565e59304cd1a21318b035529384 (127.0.98.1:45963)
I20260620 09:05:32.112259 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:32.121444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:32.232336 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:32.232573 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:32.232614 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:32.236567 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:32.236630 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:32.236685 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:32.241694 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:32.242625 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:32.243595 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:32.252950 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.252944 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.252970 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:32.253230 4171 server_base.cc:1034] running on GCE node
I20260620 09:05:32.253789 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:32.254387 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:32.255558 4171 hybrid_clock.cc:648] HybridClock initialized: now 1781946332255542 us; error 25 us; skew 500 ppm
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:32.258289 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:32.259140 4171 webserver.cc:460] Webserver started at http://127.0.98.2:36739/ using document root <none> and password file <none>
I20260620 09:05:32.259711 4171 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:32.259799 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:32.260017 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:32.262367 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "b46f887c2227462f8c15c9a419f32a42"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "16e312255fa76162a7fa53602cfa5f55"
server_key_iv: "e7fd1e272b2b4003e5244ec2a12524d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.262931 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b46f887c2227462f8c15c9a419f32a42"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "16e312255fa76162a7fa53602cfa5f55"
server_key_iv: "e7fd1e272b2b4003e5244ec2a12524d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.266460 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:05:32.268766 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.269976 4171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:32.270105 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "b46f887c2227462f8c15c9a419f32a42"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "16e312255fa76162a7fa53602cfa5f55"
server_key_iv: "e7fd1e272b2b4003e5244ec2a12524d8"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.270210 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:32.282231 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:32.285346 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:32.285537 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:32.286082 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:32.286854 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:32.286921 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.286991 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:32.287034 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.295727 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33701
I20260620 09:05:32.295745 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33701 every 8 connection(s)
I20260620 09:05:32.296594 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:32.297181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4171
I20260620 09:05:32.297292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:32.297611 392 external_mini_cluster.cc:1428] Setting key 3cc9380f758d4b488dd0794a06d0757f
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:05:32.311599 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.298527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53685 (local address 127.0.98.62:34489)
0620 09:05:32.298671 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:32.298675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:32.299781 (+ 1106us) server_negotiation.cc:407] Connection header received
0620 09:05:32.301589 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:32.301592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:32.301644 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:32.301721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:32.302837 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.303721 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.304873 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.305181 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.305294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.308012 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:32.308015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:32.308017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:32.308041 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:32.309475 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.309918 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.309920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.309922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.309992 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.310332 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.310334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.310335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.310500 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:32.310600 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:32.311202 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 09:05:32.311353 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:05:32.312232 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.299087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.2:53685)
0620 09:05:32.299601 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:32.299660 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:32.301392 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:32.301880 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:32.301886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:32.302244 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:32.302700 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.302712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.303848 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.303851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.304746 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.304755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.305331 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.305335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.305503 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.306096 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:32.306111 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:32.307846 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:32.309585 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.309590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.309601 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.309836 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.310067 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.310070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.310071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.310252 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.310591 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:32.310595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:32.310931 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 09:05:32.311133 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 09:05:32.311385 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260620 09:05:32.313515 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 09:05:32.313762 4300 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:32.314321 4300 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 09:05:32.315436 3975 ts_manager.cc:194] Registered new tserver with Master: b46f887c2227462f8c15c9a419f32a42 (127.0.98.2:33701)
I20260620 09:05:32.316319 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:32.327347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:32.435024 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:32.435261 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:32.435340 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:32.439441 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:32.439500 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:32.439564 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:32.444640 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:32.445695 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:32.446705 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:32.455971 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.456132 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:32.455984 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:32.456370 4305 server_base.cc:1034] running on GCE node
I20260620 09:05:32.456748 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:32.457396 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:32.458575 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781946332458553 us; error 39 us; skew 500 ppm
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:32.461186 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:32.461982 4305 webserver.cc:460] Webserver started at http://127.0.98.3:41569/ using document root <none> and password file <none>
I20260620 09:05:32.462543 4305 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:32.462630 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:32.462849 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:32.464850 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d17336ba5174ad0983909c40f42a3ae"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "d71b3c4b8d442b7d844974c6d1785dff"
server_key_iv: "876734f1dafa63e13663cab097d5a123"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.465472 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d17336ba5174ad0983909c40f42a3ae"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "d71b3c4b8d442b7d844974c6d1785dff"
server_key_iv: "876734f1dafa63e13663cab097d5a123"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.468942 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:32.471330 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.472520 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:32.472654 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "2d17336ba5174ad0983909c40f42a3ae"
format_stamp: "Formatted at 2026-06-20 09:05:32 on dist-test-slave-92t4"
server_key: "d71b3c4b8d442b7d844974c6d1785dff"
server_key_iv: "876734f1dafa63e13663cab097d5a123"
server_key_version: "encryptionkey@0"
I20260620 09:05:32.472759 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:32.483355 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:32.485986 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:32.486183 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:32.486727 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:32.487517 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:32.487596 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.487663 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:32.487713 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:32.497663 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35661
I20260620 09:05:32.497718 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35661 every 8 connection(s)
I20260620 09:05:32.498603 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:32.503448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4305
I20260620 09:05:32.503567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:32.503849 392 external_mini_cluster.cc:1428] Setting key fd311661a76e0157ae635eecfb5277d5
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:32.512133 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.500333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50181 (local address 127.0.98.62:34489)
0620 09:05:32.500500 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:32.500504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:32.501362 (+ 858us) server_negotiation.cc:407] Connection header received
0620 09:05:32.502666 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:32.502668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:32.502715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:32.502790 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:32.504248 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.505019 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.506340 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.506599 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.506705 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.509271 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:32.509274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:32.509276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:32.509296 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:32.510501 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.510921 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.510924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.510925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.510994 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.511269 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.511271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.511272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.511425 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:32.511497 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:32.511924 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 09:05:32.512013 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 09:05:32.512864 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.500747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.3:50181)
0620 09:05:32.501226 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:32.501268 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:32.502473 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:32.502973 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:32.502980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:32.503420 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:32.504091 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.504112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.505199 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.505204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.506239 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:32.506248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.506786 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.506792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:32.507014 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.507721 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:32.507736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:32.509136 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:32.510594 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.510598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.510610 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.510842 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.511067 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:32.511069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:32.511070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:32.511189 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:32.511505 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:32.511522 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:32.511726 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 09:05:32.511885 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 09:05:32.512108 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
I20260620 09:05:32.513909 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 09:05:32.514107 4434 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:32.514515 4434 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 09:05:32.515412 3975 ts_manager.cc:194] Registered new tserver with Master: 2d17336ba5174ad0983909c40f42a3ae (127.0.98.3:35661)
I20260620 09:05:32.515973 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50181
I20260620 09:05:32.517401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:32.541486 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.532936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.1:45963)
0620 09:05:32.533240 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:32.533254 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:32.533286 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:05:32.533336 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:32.533340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:32.533523 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:32.533644 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:32.534247 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.535009 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.535777 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.536127 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.536204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.538471 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:32.538479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:32.538491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:32.538513 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:32.539911 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.540275 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.540279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.540280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.540350 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.540567 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.540569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.540570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.540784 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:32.540930 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:32.541138 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 09:05:32.541292 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":135,"threads_started":1}
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:32.551775 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:32.544134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.0.98.62:34489)
0620 09:05:32.544253 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:32.544256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:32.544269 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:32.544346 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:32.544348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:32.544384 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:32.544462 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:32.545024 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.545573 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.546292 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:32.546492 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:32.546565 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:32.548840 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:32.548843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:32.548846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:32.548867 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:32.550145 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.550562 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.550568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.550574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.550645 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:32.550924 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:32.550928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:32.550930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:32.551141 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:32.551277 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:32.551500 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 09:05:32.551616 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 09:05:32.552170 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33198
I20260620 09:05:32.552778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4037
I20260620 09:05:32.558528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4171
I20260620 09:05:32.563747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4305
I20260620 09:05:32.568851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 3943
2026-06-20T09:05:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2067 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[4446](info): set up 2 sockets
Jun 20 09:05:32 dist-test-slave-92t4 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260620 09:05:33.597105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:33 dist-test-slave-92t4 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:33.723181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:33.828970 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:33.829282 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:33.829373 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:33.833316 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:33.833400 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:33.833437 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:33.833491 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:33.833518 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:33.838666 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:33.839658 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:33.840682 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:33.848560 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:33.848570 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:33.848569 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:33.848739 4459 server_base.cc:1034] running on GCE node
I20260620 09:05:33.849434 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:33.850467 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:33.851670 4459 hybrid_clock.cc:648] HybridClock initialized: now 1781946333851659 us; error 27 us; skew 500 ppm
Jun 20 09:05:33 dist-test-slave-92t4 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:33.854336 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:33.855011 4459 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:33.855078 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:33.855278 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:33.857257 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "c0100bfa842241e9958234c3de4d2429"
format_stamp: "Formatted at 2026-06-20 09:05:33 on dist-test-slave-92t4"
server_key: "89ffe369291fb05afe43a537d265ca02"
server_key_iv: "33e08fffe1d1ab4db110996a380e1af3"
server_key_version: "encryptionkey@0"
I20260620 09:05:33.857789 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0100bfa842241e9958234c3de4d2429"
format_stamp: "Formatted at 2026-06-20 09:05:33 on dist-test-slave-92t4"
server_key: "89ffe369291fb05afe43a537d265ca02"
server_key_iv: "33e08fffe1d1ab4db110996a380e1af3"
server_key_version: "encryptionkey@0"
I20260620 09:05:33.861351 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:33.863549 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:33.864626 4459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:33.864742 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "c0100bfa842241e9958234c3de4d2429"
format_stamp: "Formatted at 2026-06-20 09:05:33 on dist-test-slave-92t4"
server_key: "89ffe369291fb05afe43a537d265ca02"
server_key_iv: "33e08fffe1d1ab4db110996a380e1af3"
server_key_version: "encryptionkey@0"
I20260620 09:05:33.864832 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:33.896036 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:33.898866 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:33.899084 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:33.905747 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40497
I20260620 09:05:33.906004 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40497 every 8 connection(s)
I20260620 09:05:33.906409 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:33.909142 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:33.909162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4459
I20260620 09:05:33.909294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:33.909559 392 external_mini_cluster.cc:1428] Setting key a3d5c94303359a70d4698f1df84fe028
I20260620 09:05:33.916056 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: Bootstrap starting.
I20260620 09:05:33.918548 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: Neither blocks nor log segments found. Creating new log.
Jun 20 09:05:33 dist-test-slave-92t4 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:33.919375 4526 log.cc:824] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:33.921490 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: No bootstrap required, opened a new log
I20260620 09:05:33.923681 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:33.910898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.0.98.62:40497)
0620 09:05:33.911282 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:33.911303 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:33.911343 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:05:33.912443 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:33.912467 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:33.912902 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:33.913277 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:33.913862 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:33.915326 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:33.916152 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:33.916808 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:33.916954 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:33.919357 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:33.919364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:33.919378 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:33.919409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:33.921022 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:33.921512 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:33.921516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:33.921521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:33.921671 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:33.921944 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:33.921946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:33.921947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:33.922286 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:33.922418 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:33.922692 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:05:33.922978 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260620 09:05:33.925392 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260620 09:05:33.925704 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:33.925760 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0100bfa842241e9958234c3de4d2429, State: Initialized, Role: FOLLOWER
I20260620 09:05:33.926146 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260620 09:05:33.926319 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:33.926397 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:33.926515 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:33.927347 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260620 09:05:33.927668 4526 leader_election.cc:302] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0100bfa842241e9958234c3de4d2429; no voters:
I20260620 09:05:33.927954 4526 leader_election.cc:288] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:33.928092 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:33.928289 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [term 1 LEADER]: Becoming Leader. State: Replica: c0100bfa842241e9958234c3de4d2429, State: Running, Role: LEADER
I20260620 09:05:33.928606 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260620 09:05:33.929248 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:33.930240 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0100bfa842241e9958234c3de4d2429. Latest consensus state: current_term: 1 leader_uuid: "c0100bfa842241e9958234c3de4d2429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260620 09:05:33.930356 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:33.930688 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0100bfa842241e9958234c3de4d2429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100bfa842241e9958234c3de4d2429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260620 09:05:33.930779 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:33.930850 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:33.933777 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:33.938616 4540 catalog_manager.cc:1347] Generated new cluster ID: 7725613d0cf04130835e6a6377df2e95
I20260620 09:05:33.938674 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:33.954246 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:33.955466 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:33.961895 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: Generated new TSK 0
I20260620 09:05:33.962695 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:33.994282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:34.107221 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:34.107514 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:34.107560 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:34.111495 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:34.111549 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:34.111626 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:34.116729 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:34.117746 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:34.118695 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:34.127693 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.127733 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.127869 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:34.128000 4552 server_base.cc:1034] running on GCE node
I20260620 09:05:34.128410 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:34.129032 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:34.130265 4552 hybrid_clock.cc:648] HybridClock initialized: now 1781946334130239 us; error 30 us; skew 500 ppm
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:34.133422 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:34.134385 4552 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:34.134497 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:34.134737 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:34.137190 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "502406b5281d4bddb950497df8cf6c11"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "e9629617082715fbb1d3dfae84e991f8"
server_key_iv: "3da1ef8bb107f7c189754972ba6580c9"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.137940 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "502406b5281d4bddb950497df8cf6c11"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "e9629617082715fbb1d3dfae84e991f8"
server_key_iv: "3da1ef8bb107f7c189754972ba6580c9"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.142000 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:34.144240 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.145483 4552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:34.145573 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "502406b5281d4bddb950497df8cf6c11"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "e9629617082715fbb1d3dfae84e991f8"
server_key_iv: "3da1ef8bb107f7c189754972ba6580c9"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.145664 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:34.167291 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:34.170781 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:34.170984 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:34.171583 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:34.172431 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:34.172501 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.172578 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:34.172626 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.181443 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41153
I20260620 09:05:34.181463 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41153 every 8 connection(s)
I20260620 09:05:34.182036 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:34.190438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4552
I20260620 09:05:34.190570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:34.190871 392 external_mini_cluster.cc:1428] Setting key c348bc3d220d3fd19bf9f584aec3bbd2
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:34.195880 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.183718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54429 (local address 127.0.98.62:40497)
0620 09:05:34.183853 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.183857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.184654 (+ 797us) server_negotiation.cc:407] Connection header received
0620 09:05:34.186040 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.186045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.186102 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.186210 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:34.187346 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.187904 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.189161 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.189384 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.189499 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.192175 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:34.192179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:34.192183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:34.192212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:34.193778 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.194308 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.194313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.194315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.194398 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.194780 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.194783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.194784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.194968 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:34.195046 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.195661 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.195757 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:05:34.197175 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.184077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.1:54429)
0620 09:05:34.184501 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.184541 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.185824 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.186375 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.186382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.186739 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:34.187197 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.187210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.188027 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.188031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.188974 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.188983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.189531 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.189535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.189717 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.190331 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:34.190352 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:34.191995 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:34.193913 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.193919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.193934 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.194203 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.194496 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.194499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.194501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.194666 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.195074 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:34.195084 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:34.195401 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.195626 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.195995 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260620 09:05:34.198418 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260620 09:05:34.198647 4680 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:34.199077 4680 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:34.200511 4490 ts_manager.cc:194] Registered new tserver with Master: 502406b5281d4bddb950497df8cf6c11 (127.0.98.1:41153)
I20260620 09:05:34.201694 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:34.217377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:34.324653 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:34.324923 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:34.324996 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:34.328982 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:34.329072 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:34.329180 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:34.334642 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:34.335654 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:34.336642 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:34.345831 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.345822 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.345820 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:34.346227 4685 server_base.cc:1034] running on GCE node
I20260620 09:05:34.346616 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:34.347237 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:34.348417 4685 hybrid_clock.cc:648] HybridClock initialized: now 1781946334348400 us; error 30 us; skew 500 ppm
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:34.351138 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:34.351866 4685 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:34.351958 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:34.352185 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:34.354313 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "be55960dde684eb98d5453e6310e1781"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "ebd3f4544cd31d57e184da39aabc9c37"
server_key_iv: "940c6fed0f988a0b2123ea318eba6369"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.354857 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be55960dde684eb98d5453e6310e1781"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "ebd3f4544cd31d57e184da39aabc9c37"
server_key_iv: "940c6fed0f988a0b2123ea318eba6369"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.358392 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:34.360684 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.361920 4685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:34.362046 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "be55960dde684eb98d5453e6310e1781"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "ebd3f4544cd31d57e184da39aabc9c37"
server_key_iv: "940c6fed0f988a0b2123ea318eba6369"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.362177 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:34.377769 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:34.380399 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:34.380590 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:34.381240 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:34.382001 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:34.382079 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.382157 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:34.382202 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.390570 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38603
I20260620 09:05:34.390596 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38603 every 8 connection(s)
I20260620 09:05:34.391141 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:34.393172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4685
I20260620 09:05:34.393309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:34.393589 392 external_mini_cluster.cc:1428] Setting key c1f9de7e66f9377dcbaef0138096b61d
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:34.405323 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.392790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37663 (local address 127.0.98.62:40497)
0620 09:05:34.392936 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.392943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.393835 (+ 892us) server_negotiation.cc:407] Connection header received
0620 09:05:34.395349 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.395352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.395401 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.395505 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:34.396619 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.397226 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.398481 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.398910 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.399055 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.401674 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:34.401678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:34.401682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:34.401716 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:34.403362 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.403832 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.403835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.403837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.403909 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.404199 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.404201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.404202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.404428 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:34.404509 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.404965 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.405158 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:05:34.406246 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.393184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.2:37663)
0620 09:05:34.393663 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.393716 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.395149 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.395622 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.395628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.395962 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:34.396429 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.396441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.397338 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.397343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.398343 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.398351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.399074 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.399077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.399209 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.399815 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:34.399831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:34.401496 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:34.403481 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.403485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.403498 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.403743 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.403989 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.403991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.403993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.404123 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.404519 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:34.404524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:34.404764 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.404923 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.405280 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260620 09:05:34.407469 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260620 09:05:34.407732 4813 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:34.408162 4813 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
I20260620 09:05:34.409222 4490 ts_manager.cc:194] Registered new tserver with Master: be55960dde684eb98d5453e6310e1781 (127.0.98.2:38603)
I20260620 09:05:34.409804 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:34.419972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:34.531826 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:34.532078 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:34.532150 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:34.536186 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:34.536235 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:34.536334 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:34.541481 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:34.542692 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:34.543676 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:34.553051 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.553164 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:34.553122 4818 server_base.cc:1034] running on GCE node
W20260620 09:05:34.553084 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:34.553699 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:34.554356 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:34.555529 4818 hybrid_clock.cc:648] HybridClock initialized: now 1781946334555504 us; error 46 us; skew 500 ppm
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:34.558213 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:34.558961 4818 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:34.559059 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:34.559283 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:34.561409 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "109d2c6a0d0748969131bfdafb7513c9"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "c57d9c5a378e313245e7e71021051c4d"
server_key_iv: "abb77c5e76e2b04c58c3d85433854e37"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.561935 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "109d2c6a0d0748969131bfdafb7513c9"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "c57d9c5a378e313245e7e71021051c4d"
server_key_iv: "abb77c5e76e2b04c58c3d85433854e37"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.565474 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:34.567782 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.568976 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:34.569135 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "109d2c6a0d0748969131bfdafb7513c9"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "c57d9c5a378e313245e7e71021051c4d"
server_key_iv: "abb77c5e76e2b04c58c3d85433854e37"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.569242 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:34.579584 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:34.582283 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:34.582474 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:34.583042 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:34.583812 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:34.583881 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.583949 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:34.584005 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.592437 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37201
I20260620 09:05:34.592460 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37201 every 8 connection(s)
I20260620 09:05:34.592991 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:34.596107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4818
I20260620 09:05:34.596244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:34.596583 392 external_mini_cluster.cc:1428] Setting key ef57b6701da41b186fcdcd3a0b2f3667
Jun 20 09:05:34 dist-test-slave-92t4 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:34.605996 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.594636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50397 (local address 127.0.98.62:40497)
0620 09:05:34.594797 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.594801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.595596 (+ 795us) server_negotiation.cc:407] Connection header received
0620 09:05:34.597318 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.597321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.597375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.597477 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:34.598903 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.599421 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.600495 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.600698 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.600771 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.603089 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:34.603092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:34.603094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:34.603114 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:34.604298 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.604721 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.604724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.604725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.604793 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.605129 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.605132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.605133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.605291 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:34.605364 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.605785 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.605875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 09:05:34.606774 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.595010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.3:50397)
0620 09:05:34.595433 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.595491 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.597097 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.597711 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.597720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.598188 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:34.598782 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.598797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.599518 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.599520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.600400 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.600408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.600815 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.600817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.600946 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.601576 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:34.601591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:34.602957 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:34.604392 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.604396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.604406 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.604643 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.604866 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:34.604868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:34.604869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:34.604986 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:34.605375 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:34.605379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:34.605593 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.605741 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.605969 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260620 09:05:34.607857 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260620 09:05:34.608114 4946 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:34.608546 4946 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
I20260620 09:05:34.609546 4489 ts_manager.cc:194] Registered new tserver with Master: 109d2c6a0d0748969131bfdafb7513c9 (127.0.98.3:37201)
I20260620 09:05:34.610263 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:05:34.610335 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50397
I20260620 09:05:34.618471 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.612165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.0.98.62:40497)
0620 09:05:34.612292 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.612296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.612429 (+ 133us) server_negotiation.cc:407] Connection header received
0620 09:05:34.612520 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.612522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.612555 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.612618 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:34.613305 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.613779 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.614516 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.614702 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.614769 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.615937 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:34.615940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:34.615942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:34.615963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:34.617197 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.617542 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.617544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.617545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.617609 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:34.617814 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:34.617816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:34.617817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:34.618021 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:34.618117 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.618282 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.618365 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:05:34.621030 4489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:34.623961 4489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:34.637301 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.631750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38603 (local address 127.0.0.1:56368)
0620 09:05:34.632421 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.632469 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.632589 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.633395 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.633400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.633421 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:34.633539 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.633545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.634748 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.634753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.635973 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.635981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.636773 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.636777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.636876 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.636912 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.637025 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.637156 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260620 09:05:34.637339 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.631898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.98.2:38603)
0620 09:05:34.632877 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.632889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.632912 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:05:34.632988 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.632994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.633250 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.633396 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:34.633650 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.634610 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.636093 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.636665 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.636768 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.636900 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.637095 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.637193 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":160,"threads_started":1}
I20260620 09:05:34.638088 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.631854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37201 (local address 127.0.0.1:49408)
0620 09:05:34.633113 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.633126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.633233 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.633730 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.633733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.633749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:34.633826 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.633831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.635184 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.635187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.636457 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.636466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.637718 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.637721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.637808 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.637828 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.637897 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.637961 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
I20260620 09:05:34.638418 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.632101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41153 (local address 127.0.0.1:33074)
0620 09:05:34.633185 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.633198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.633291 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.634019 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.634022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.634038 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:34.634121 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.634126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.635395 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.635398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.636569 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.636578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.637904 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.637907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.638006 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.638027 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.638101 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.638297 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":25,"thread_start_us":78,"threads_started":1}
I20260620 09:05:34.638523 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.632514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.0.98.3:37201)
0620 09:05:34.632942 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.632955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.633211 (+ 256us) server_negotiation.cc:407] Connection header received
0620 09:05:34.633382 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.633387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.633575 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.633730 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:34.633918 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.635047 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.636603 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.637407 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.637516 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.637657 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.638262 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.638363 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
I20260620 09:05:34.640183 4747 tablet_service.cc:1467] Processing CreateTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 (DEFAULT_TABLE table=test-table [id=b4e92a0bafd44d2eb25d7f08f49a00ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:34.640654 4880 tablet_service.cc:1467] Processing CreateTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 (DEFAULT_TABLE table=test-table [id=b4e92a0bafd44d2eb25d7f08f49a00ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:34.641106 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08046da605ee4d2d9ed91d7a072b2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:34.641644 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08046da605ee4d2d9ed91d7a072b2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:34.645656 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.632522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.98.1:41153)
0620 09:05:34.632959 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.632973 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.633277 (+ 304us) server_negotiation.cc:407] Connection header received
0620 09:05:34.633726 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.633731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.633889 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.634034 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:34.634211 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.635272 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.636654 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.637787 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.637903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.638027 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.638124 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.645505 (+ 7381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":99,"threads_started":1}
I20260620 09:05:34.647697 4964 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Bootstrap starting.
I20260620 09:05:34.648389 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 (DEFAULT_TABLE table=test-table [id=b4e92a0bafd44d2eb25d7f08f49a00ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:34.649343 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08046da605ee4d2d9ed91d7a072b2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:34.649349 4963 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Bootstrap starting.
I20260620 09:05:34.649953 4964 tablet_bootstrap.cc:654] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:34.651223 4964 log.cc:824] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:34.651571 4963 tablet_bootstrap.cc:654] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:34.652691 4963 log.cc:824] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:34.653326 4964 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: No bootstrap required, opened a new log
I20260620 09:05:34.653535 4964 ts_tablet_manager.cc:1397] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:34.654644 4963 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: No bootstrap required, opened a new log
I20260620 09:05:34.654821 4963 ts_tablet_manager.cc:1397] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:05:34.655429 4967 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Bootstrap starting.
I20260620 09:05:34.657302 4967 tablet_bootstrap.cc:654] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:34.657402 4964 raft_consensus.cc:348] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.657670 4964 raft_consensus.cc:374] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:34.657725 4964 raft_consensus.cc:729] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be55960dde684eb98d5453e6310e1781, State: Initialized, Role: FOLLOWER
I20260620 09:05:34.658167 4964 consensus_queue.cc:260] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.658455 4967 log.cc:824] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:34.658543 4963 raft_consensus.cc:348] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.658780 4963 raft_consensus.cc:374] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:34.658824 4963 raft_consensus.cc:729] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109d2c6a0d0748969131bfdafb7513c9, State: Initialized, Role: FOLLOWER
I20260620 09:05:34.658855 4813 heartbeater.cc:503] Master 127.0.98.62:40497 was elected leader, sending a full tablet report...
I20260620 09:05:34.659171 4964 ts_tablet_manager.cc:1428] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:34.659250 4963 consensus_queue.cc:260] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.660190 4963 ts_tablet_manager.cc:1428] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 09:05:34.660750 4946 heartbeater.cc:503] Master 127.0.98.62:40497 was elected leader, sending a full tablet report...
I20260620 09:05:34.660929 4967 tablet_bootstrap.cc:492] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: No bootstrap required, opened a new log
I20260620 09:05:34.661157 4967 ts_tablet_manager.cc:1397] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 09:05:34.664551 4967 raft_consensus.cc:348] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.664767 4967 raft_consensus.cc:374] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:34.664829 4967 raft_consensus.cc:729] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502406b5281d4bddb950497df8cf6c11, State: Initialized, Role: FOLLOWER
I20260620 09:05:34.665304 4967 consensus_queue.cc:260] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.665875 4680 heartbeater.cc:503] Master 127.0.98.62:40497 was elected leader, sending a full tablet report...
I20260620 09:05:34.666108 4967 ts_tablet_manager.cc:1428] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260620 09:05:34.683077 4681 tablet.cc:2369] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:34.708775 4970 raft_consensus.cc:482] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:34.709000 4970 raft_consensus.cc:504] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.710207 4970 leader_election.cc:288] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be55960dde684eb98d5453e6310e1781 (127.0.98.2:38603), 502406b5281d4bddb950497df8cf6c11 (127.0.98.1:41153)
I20260620 09:05:34.714121 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.710496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36185 (local address 127.0.98.2:38603)
0620 09:05:34.710630 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.710637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.710657 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:05:34.710823 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.710828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.710886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.710965 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:34.711242 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.711796 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.713000 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.713604 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.713681 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.713731 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.713942 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.714003 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:34.714140 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.710407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38603 (local address 127.0.98.3:36185)
0620 09:05:34.710546 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.710563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.710653 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.710995 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.710998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.711014 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:34.711133 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.711138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.711919 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.711922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.712876 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.712884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.713714 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.713717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.713822 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.713846 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.713925 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.713992 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260620 09:05:34.714627 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.710562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45479 (local address 127.0.98.1:41153)
0620 09:05:34.710728 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.710733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.711007 (+ 274us) server_negotiation.cc:407] Connection header received
0620 09:05:34.711174 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.711177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.711227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.711309 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:34.711574 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.712295 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.713496 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.714038 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.714142 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.714211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.714454 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.714525 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 09:05:34.714648 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.710424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41153 (local address 127.0.98.3:45479)
0620 09:05:34.710908 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.710919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:34.711060 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:34.711321 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:34.711324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:34.711339 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:34.711463 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.711468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.712405 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.712407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.713380 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:34.713390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.714178 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.714182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:34.714279 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.714312 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 09:05:34.714434 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 09:05:34.714504 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":114,"spinlock_wait_cycles":27520,"thread_start_us":132,"threads_started":1}
I20260620 09:05:34.714908 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08046da605ee4d2d9ed91d7a072b2498" candidate_uuid: "109d2c6a0d0748969131bfdafb7513c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be55960dde684eb98d5453e6310e1781" is_pre_election: true
I20260620 09:05:34.715238 4767 raft_consensus.cc:2454] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 109d2c6a0d0748969131bfdafb7513c9 in term 0.
I20260620 09:05:34.715242 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08046da605ee4d2d9ed91d7a072b2498" candidate_uuid: "109d2c6a0d0748969131bfdafb7513c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502406b5281d4bddb950497df8cf6c11" is_pre_election: true
I20260620 09:05:34.715533 4634 raft_consensus.cc:2454] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 109d2c6a0d0748969131bfdafb7513c9 in term 0.
I20260620 09:05:34.715744 4834 leader_election.cc:302] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109d2c6a0d0748969131bfdafb7513c9, be55960dde684eb98d5453e6310e1781; no voters:
I20260620 09:05:34.716032 4970 raft_consensus.cc:2781] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:34.716109 4970 raft_consensus.cc:482] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:34.716138 4970 raft_consensus.cc:3037] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:34.717211 4970 raft_consensus.cc:504] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.717554 4970 leader_election.cc:288] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [CANDIDATE]: Term 1 election: Requested vote from peers be55960dde684eb98d5453e6310e1781 (127.0.98.2:38603), 502406b5281d4bddb950497df8cf6c11 (127.0.98.1:41153)
I20260620 09:05:34.717859 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08046da605ee4d2d9ed91d7a072b2498" candidate_uuid: "109d2c6a0d0748969131bfdafb7513c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502406b5281d4bddb950497df8cf6c11"
I20260620 09:05:34.717878 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08046da605ee4d2d9ed91d7a072b2498" candidate_uuid: "109d2c6a0d0748969131bfdafb7513c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be55960dde684eb98d5453e6310e1781"
I20260620 09:05:34.717996 4634 raft_consensus.cc:3037] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:34.718022 4767 raft_consensus.cc:3037] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:34.719002 4634 raft_consensus.cc:2454] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109d2c6a0d0748969131bfdafb7513c9 in term 1.
I20260620 09:05:34.719115 4767 raft_consensus.cc:2454] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109d2c6a0d0748969131bfdafb7513c9 in term 1.
I20260620 09:05:34.719367 4833 leader_election.cc:302] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109d2c6a0d0748969131bfdafb7513c9, 502406b5281d4bddb950497df8cf6c11; no voters:
I20260620 09:05:34.719518 4970 raft_consensus.cc:2781] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:34.719732 4970 raft_consensus.cc:686] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 1 LEADER]: Becoming Leader. State: Replica: 109d2c6a0d0748969131bfdafb7513c9, State: Running, Role: LEADER
I20260620 09:05:34.720038 4970 consensus_queue.cc:237] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } }
I20260620 09:05:34.723003 4489 catalog_manager.cc:5495] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 109d2c6a0d0748969131bfdafb7513c9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "109d2c6a0d0748969131bfdafb7513c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "109d2c6a0d0748969131bfdafb7513c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 } health_report { overall_health: UNKNOWN } } }
I20260620 09:05:34.733594 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.730042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.0.98.3:37201)
0620 09:05:34.730196 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.730200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.730214 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:34.730366 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.730369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.730432 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.730537 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:34.730765 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.731315 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.732150 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:34.732379 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:34.732486 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:34.732761 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:34.733225 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:34.733347 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.733407 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.733456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:34.738866 4767 raft_consensus.cc:1261] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 1 FOLLOWER]: Refusing update from remote peer 109d2c6a0d0748969131bfdafb7513c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:34.738866 4634 raft_consensus.cc:1261] T 08046da605ee4d2d9ed91d7a072b2498 P 502406b5281d4bddb950497df8cf6c11 [term 1 FOLLOWER]: Refusing update from remote peer 109d2c6a0d0748969131bfdafb7513c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:34.739490 4973 consensus_queue.cc:1035] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502406b5281d4bddb950497df8cf6c11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:34.739686 4970 consensus_queue.cc:1035] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55960dde684eb98d5453e6310e1781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:34.747972 4977 mvcc.cc:205] Tried to move back new op lower bound from 7298852187084034048 to 7298852187015716864. Current Snapshot: MvccSnapshot[applied={T|T < 7298852187084034048}]
I20260620 09:05:34.755689 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:34.755829 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:34.757983 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0100bfa842241e9958234c3de4d2429: Sending DeleteTablet for 3 replicas of tablet 08046da605ee4d2d9ed91d7a072b2498
I20260620 09:05:34.758697 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49408
I20260620 09:05:34.758713 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56368
I20260620 09:05:34.759145 4986 tablet_replica.cc:330] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: stopping tablet replica
I20260620 09:05:34.759303 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 08046da605ee4d2d9ed91d7a072b2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33074
I20260620 09:05:34.759559 4986 raft_consensus.cc:2229] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:34.759740 4987 tablet_replica.cc:330] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: stopping tablet replica
I20260620 09:05:34.759790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4552
I20260620 09:05:34.759999 4987 raft_consensus.cc:2229] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:34.761422 4987 raft_consensus.cc:2258] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:34.761636 4986 raft_consensus.cc:2258] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:34.762987 4986 ts_tablet_manager.cc:1905] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:34.762987 4987 ts_tablet_manager.cc:1905] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:34.765647 4986 ts_tablet_manager.cc:1918] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:34.765705 4987 ts_tablet_manager.cc:1918] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:34.765790 4987 log.cc:1196] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/08046da605ee4d2d9ed91d7a072b2498
I20260620 09:05:34.765823 4986 log.cc:1196] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/08046da605ee4d2d9ed91d7a072b2498
I20260620 09:05:34.766137 4987 ts_tablet_manager.cc:1939] T 08046da605ee4d2d9ed91d7a072b2498 P be55960dde684eb98d5453e6310e1781: Deleting consensus metadata
I20260620 09:05:34.766321 4986 ts_tablet_manager.cc:1939] T 08046da605ee4d2d9ed91d7a072b2498 P 109d2c6a0d0748969131bfdafb7513c9: Deleting consensus metadata
I20260620 09:05:34.767345 4475 catalog_manager.cc:4841] TS be55960dde684eb98d5453e6310e1781 (127.0.98.2:38603): tablet 08046da605ee4d2d9ed91d7a072b2498 (table test-table [id=b4e92a0bafd44d2eb25d7f08f49a00ea]) successfully deleted
I20260620 09:05:34.767345 4476 catalog_manager.cc:4841] TS 109d2c6a0d0748969131bfdafb7513c9 (127.0.98.3:37201): tablet 08046da605ee4d2d9ed91d7a072b2498 (table test-table [id=b4e92a0bafd44d2eb25d7f08f49a00ea]) successfully deleted
W20260620 09:05:34.768407 4474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41153 (error 108)
I20260620 09:05:34.768600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4685
I20260620 09:05:34.770015 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.769753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41153 (local address 127.0.0.1:33080)
0620 09:05:34.769886 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:34.769949 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:05:34.770205 4474 catalog_manager.cc:4584] TS 502406b5281d4bddb950497df8cf6c11 (127.0.98.1:41153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08046da605ee4d2d9ed91d7a072b2498: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41153: connect: Connection refused (error 111)
I20260620 09:05:34.775884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4818
I20260620 09:05:34.780438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4459
2026-06-20T09:05:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2211 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T09:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:34Z Disabled control of system clock
I20260620 09:05:34.808684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38993
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:05:34.914386 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:34.914624 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:34.914681 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:34.918514 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:34.918558 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:34.918574 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:34.918589 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:34.918604 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:34.923156 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:34.924199 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:34.925606 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:34.933947 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.933959 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:34.933969 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:34.934304 4993 server_base.cc:1034] running on GCE node
I20260620 09:05:34.934870 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:34.935920 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:34.937114 4993 hybrid_clock.cc:648] HybridClock initialized: now 1781946334937100 us; error 26 us; skew 500 ppm
I20260620 09:05:34.938849 4993 webserver.cc:460] Webserver started at http://127.0.98.62:39845/ using document root <none> and password file <none>
I20260620 09:05:34.939414 4993 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:34.939497 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:34.939725 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:34.941788 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "449863d8cc634163bdecbfba2f903272"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "76ade2ee1d4eade2b061fe9dd17e701e"
server_key_iv: "6d57891a8395fc393af1a2cda77c315d"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.942349 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "449863d8cc634163bdecbfba2f903272"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "76ade2ee1d4eade2b061fe9dd17e701e"
server_key_iv: "6d57891a8395fc393af1a2cda77c315d"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.946122 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:34.948494 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:34.949811 4993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:34.949947 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "449863d8cc634163bdecbfba2f903272"
format_stamp: "Formatted at 2026-06-20 09:05:34 on dist-test-slave-92t4"
server_key: "76ade2ee1d4eade2b061fe9dd17e701e"
server_key_iv: "6d57891a8395fc393af1a2cda77c315d"
server_key_version: "encryptionkey@0"
I20260620 09:05:34.950060 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:34.954703 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:34.955317 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:34.955498 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:34.962026 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38993
I20260620 09:05:34.962052 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38993 every 8 connection(s)
I20260620 09:05:34.962911 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:34.964077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 4993
I20260620 09:05:34.964196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:34.964459 392 external_mini_cluster.cc:1428] Setting key 5c87c8c4376487c89a4bd4b7fb545a34
I20260620 09:05:34.966042 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:34.970285 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:34.965803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.0.98.62:38993)
0620 09:05:34.966870 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:34.966884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:34.966919 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:34.967550 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:34.967572 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:34.967892 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:34.968163 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:34.968984 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:34.968988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:34.968999 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:34.969008 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:34.969112 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:34.969129 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:34.969236 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:34.969344 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:05:34.969631 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":595,"threads_started":1}
I20260620 09:05:34.972210 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: Bootstrap starting.
I20260620 09:05:34.974320 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:34.975100 5060 log.cc:824] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:34.976852 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: No bootstrap required, opened a new log
I20260620 09:05:34.980347 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } }
I20260620 09:05:34.980548 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:34.980581 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449863d8cc634163bdecbfba2f903272, State: Initialized, Role: FOLLOWER
I20260620 09:05:34.980937 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } }
I20260620 09:05:34.981092 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:34.981153 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:34.981223 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:34.981977 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } }
I20260620 09:05:34.982256 5060 leader_election.cc:302] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449863d8cc634163bdecbfba2f903272; no voters:
I20260620 09:05:34.982576 5060 leader_election.cc:288] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:34.982669 5065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:34.982896 5065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [term 1 LEADER]: Becoming Leader. State: Replica: 449863d8cc634163bdecbfba2f903272, State: Running, Role: LEADER
I20260620 09:05:34.983224 5065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } }
I20260620 09:05:34.983815 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:34.984879 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449863d8cc634163bdecbfba2f903272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } } }
I20260620 09:05:34.985003 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:34.985292 5067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449863d8cc634163bdecbfba2f903272. Latest consensus state: current_term: 1 leader_uuid: "449863d8cc634163bdecbfba2f903272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449863d8cc634163bdecbfba2f903272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38993 } } }
I20260620 09:05:34.985397 5067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:34.985498 5074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:34.988502 5074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:34.993827 5074 catalog_manager.cc:1347] Generated new cluster ID: a5730bd3a3194a9bb11bf11b5f2bac30
I20260620 09:05:34.993899 5074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:35.015592 5074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:35.015959 5074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:35.024641 5074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: Generated new TSK 0
I20260620 09:05:35.025498 5074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:05:35.038105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--builtin_ntp_servers=127.0.98.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:05:35.148248 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:35.148536 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:35.148622 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:35.152714 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:35.152773 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:35.152878 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:35.158012 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:35.159051 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:35.160351 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:35.169718 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:35.169726 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:35.169734 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:35.170065 5084 server_base.cc:1034] running on GCE node
I20260620 09:05:35.170445 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:35.171000 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:35.172154 5084 hybrid_clock.cc:648] HybridClock initialized: now 1781946335172142 us; error 24 us; skew 500 ppm
I20260620 09:05:35.174006 5084 webserver.cc:460] Webserver started at http://127.0.98.1:43189/ using document root <none> and password file <none>
I20260620 09:05:35.174602 5084 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:35.174671 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:35.174926 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:35.176970 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f956e6d5f5541478ca5dd52aa20a6c2"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "b60394c427cf0a4e73ad3b041beb2ab6"
server_key_iv: "1e31cfed7b301b9f314da85dc5eff9cf"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.177582 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f956e6d5f5541478ca5dd52aa20a6c2"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "b60394c427cf0a4e73ad3b041beb2ab6"
server_key_iv: "1e31cfed7b301b9f314da85dc5eff9cf"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.181183 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:35.183540 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.184880 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:35.185014 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "9f956e6d5f5541478ca5dd52aa20a6c2"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "b60394c427cf0a4e73ad3b041beb2ab6"
server_key_iv: "1e31cfed7b301b9f314da85dc5eff9cf"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.185200 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:35.189749 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:35.190326 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:35.190503 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:35.191032 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:35.191903 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:35.191979 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.192051 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:35.192087 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.200835 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37733
I20260620 09:05:35.200850 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37733 every 8 connection(s)
I20260620 09:05:35.201776 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:35.204258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5084
I20260620 09:05:35.204393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:35.204664 392 external_mini_cluster.cc:1428] Setting key 9c29beee0de520645987112e31c1009c
I20260620 09:05:35.206559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--builtin_ntp_servers=127.0.98.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 09:05:35.208225 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.203531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54017 (local address 127.0.98.62:38993)
0620 09:05:35.203673 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.203677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.204754 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 09:05:35.206279 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.206283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.206322 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.206396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.207646 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.207649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.207652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.207665 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.207687 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.207700 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.207795 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.208004 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.208051 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:05:35.208974 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.203869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38993 (local address 127.0.98.1:54017)
0620 09:05:35.204580 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.204641 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.206055 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.206569 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.206577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.206713 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.207373 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.207374 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.207427 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.207492 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.207811 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.207824 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.207831 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.208009 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.208195 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":156,"threads_started":1}
I20260620 09:05:35.210261 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38993
I20260620 09:05:35.210479 5212 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:35.210875 5212 heartbeater.cc:511] Master 127.0.98.62:38993 requested a full tablet report, sending...
I20260620 09:05:35.212337 5024 ts_manager.cc:194] Registered new tserver with Master: 9f956e6d5f5541478ca5dd52aa20a6c2 (127.0.98.1:37733)
W20260620 09:05:35.313908 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:35.314160 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:35.314214 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:35.318184 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:35.318229 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:35.318341 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:35.322849 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:35.323963 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:35.325356 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:35.334725 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:35.334748 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:35.334920 5215 server_base.cc:1034] running on GCE node
W20260620 09:05:35.334753 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:35.335438 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:35.336068 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:35.337239 5215 hybrid_clock.cc:648] HybridClock initialized: now 1781946335337205 us; error 45 us; skew 500 ppm
I20260620 09:05:35.338873 5215 webserver.cc:460] Webserver started at http://127.0.98.2:40139/ using document root <none> and password file <none>
I20260620 09:05:35.339437 5215 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:35.339522 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:35.339751 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:35.341806 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "c31fdbfdf46749b2bb3242eed5914bc8"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "f424c595c5fffeb7b064202052e5f6a3"
server_key_iv: "26d8b68f924b9434a0a4e918ed4188b2"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.342331 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c31fdbfdf46749b2bb3242eed5914bc8"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "f424c595c5fffeb7b064202052e5f6a3"
server_key_iv: "26d8b68f924b9434a0a4e918ed4188b2"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.345919 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:35.348165 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.349391 5215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:35.349519 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "c31fdbfdf46749b2bb3242eed5914bc8"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "f424c595c5fffeb7b064202052e5f6a3"
server_key_iv: "26d8b68f924b9434a0a4e918ed4188b2"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.349637 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:35.354197 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:35.354851 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:35.355027 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:35.355572 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:35.356438 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:35.356509 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.356581 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:35.356630 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.365116 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44741
I20260620 09:05:35.365150 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44741 every 8 connection(s)
I20260620 09:05:35.365978 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:35.371873 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.367727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58945 (local address 127.0.98.62:38993)
0620 09:05:35.367856 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.367859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.368827 (+ 968us) server_negotiation.cc:407] Connection header received
0620 09:05:35.370183 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.370187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.370232 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.370334 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.371410 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.371413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.371416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.371428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.371449 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.371463 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.371553 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.371701 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.371745 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:05:35.372673 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.368068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38993 (local address 127.0.98.2:58945)
0620 09:05:35.368682 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.368720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.369960 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.370496 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.370503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.370585 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.371162 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.371163 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.371199 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.371263 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.371547 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.371554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.371561 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.371730 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.371954 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260620 09:05:35.373282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5215
I20260620 09:05:35.373407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:35.373700 392 external_mini_cluster.cc:1428] Setting key de0eefbfefd5d49d9a4e0a0a78cfdc89
I20260620 09:05:35.373843 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38993
I20260620 09:05:35.374058 5343 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:35.374536 5343 heartbeater.cc:511] Master 127.0.98.62:38993 requested a full tablet report, sending...
I20260620 09:05:35.375602 5024 ts_manager.cc:194] Registered new tserver with Master: c31fdbfdf46749b2bb3242eed5914bc8 (127.0.98.2:44741)
I20260620 09:05:35.375640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--builtin_ntp_servers=127.0.98.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 09:05:35.493021 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:35.493319 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:35.493361 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:35.497350 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:35.497403 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:35.497463 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:35.502116 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:35.503175 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:35.504907 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:35.515301 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:35.515319 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:35.515311 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:35.515754 5346 server_base.cc:1034] running on GCE node
I20260620 09:05:35.516182 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:35.516873 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:35.518064 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781946335518053 us; error 53 us; skew 500 ppm
I20260620 09:05:35.520117 5346 webserver.cc:460] Webserver started at http://127.0.98.3:41109/ using document root <none> and password file <none>
I20260620 09:05:35.520706 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:35.520802 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:35.521116 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:35.523285 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "23dbf9b2109d44de886d8921a891540c"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "924827b0485fd5e584af526af11238bd"
server_key_iv: "e6bb7b08c5c59f5d9df3e6b48fca5f8a"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.523890 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23dbf9b2109d44de886d8921a891540c"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "924827b0485fd5e584af526af11238bd"
server_key_iv: "e6bb7b08c5c59f5d9df3e6b48fca5f8a"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.528033 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:05:35.530514 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.531816 5346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:35.531970 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "23dbf9b2109d44de886d8921a891540c"
format_stamp: "Formatted at 2026-06-20 09:05:35 on dist-test-slave-92t4"
server_key: "924827b0485fd5e584af526af11238bd"
server_key_iv: "e6bb7b08c5c59f5d9df3e6b48fca5f8a"
server_key_version: "encryptionkey@0"
I20260620 09:05:35.532081 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:35.536705 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:35.537365 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:35.537544 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:35.538121 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:35.539018 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:35.539109 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.539194 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:35.539243 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:35.548218 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42403
I20260620 09:05:35.548239 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42403 every 8 connection(s)
I20260620 09:05:35.549170 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:35.551877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5346
I20260620 09:05:35.551990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:35.552291 392 external_mini_cluster.cc:1428] Setting key b8620d9a6275ffcfae857840db381297
I20260620 09:05:35.555806 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.550926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48337 (local address 127.0.98.62:38993)
0620 09:05:35.551069 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.551074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.552378 (+ 1304us) server_negotiation.cc:407] Connection header received
0620 09:05:35.554029 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.554033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.554074 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.554184 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.555309 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.555312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.555314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.555356 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.555383 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.555402 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.555500 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.555664 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.555704 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:35.556636 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.551407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38993 (local address 127.0.98.3:48337)
0620 09:05:35.552177 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.552240 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.553838 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.554364 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.554373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.554494 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.555053 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.555054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.555094 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.555161 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.555498 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.555504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.555508 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.555651 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.555837 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260620 09:05:35.558024 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38993
I20260620 09:05:35.558243 5474 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:35.558652 5474 heartbeater.cc:511] Master 127.0.98.62:38993 requested a full tablet report, sending...
I20260620 09:05:35.559548 5024 ts_manager.cc:194] Registered new tserver with Master: 23dbf9b2109d44de886d8921a891540c (127.0.98.3:42403)
I20260620 09:05:35.566084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:05:35.569306 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.567949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.62:38993)
0620 09:05:35.568062 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.568065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.568220 (+ 155us) server_negotiation.cc:407] Connection header received
0620 09:05:35.568314 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.568316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.568345 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.568420 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.568958 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.568961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.568963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.568972 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.568990 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.568998 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.569140 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.569183 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.569213 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:35.571802 5024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:35.574941 5024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:35.586697 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.583247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44741 (local address 127.0.0.1:36816)
0620 09:05:35.584057 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.584108 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.584229 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.585167 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.585172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.585230 (+ 58us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.585849 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.585852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.585874 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.585954 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.586347 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.586359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.586367 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.586484 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.586551 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":78,"threads_started":1}
I20260620 09:05:35.586962 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.583333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.98.2:44741)
0620 09:05:35.583834 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.583847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.584269 (+ 422us) server_negotiation.cc:407] Connection header received
0620 09:05:35.584338 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.584344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.584569 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.584733 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.586111 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.586116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.586129 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.586143 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.586190 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.586229 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.586343 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.586533 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.586614 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":147,"threads_started":1}
I20260620 09:05:35.587170 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.583320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37733 (local address 127.0.0.1:41598)
0620 09:05:35.584198 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.584209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.584328 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.586043 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.586055 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.586069 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.586465 (+ 396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.586468 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.586478 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.586504 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.586906 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.586909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.586912 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.587005 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.587042 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":89,"threads_started":1}
I20260620 09:05:35.587450 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.583718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.98.1:37733)
0620 09:05:35.584037 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.584061 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.584274 (+ 213us) server_negotiation.cc:407] Connection header received
0620 09:05:35.584443 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.584450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.584650 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.584767 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.586641 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.586646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.586656 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.586668 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.586709 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.586747 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.586834 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.587111 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.587179 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
I20260620 09:05:35.587477 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.584532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.0.98.3:42403)
0620 09:05:35.584954 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.584964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.584985 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:05:35.585037 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.585043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.585246 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.585376 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.586648 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.586654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.586665 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.586675 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.586715 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.586742 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.586831 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.587235 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.587303 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":132,"threads_started":1}
I20260620 09:05:35.587705 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.584348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42403 (local address 127.0.0.1:54792)
0620 09:05:35.584638 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.584653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.584717 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.585385 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.585391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.585408 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.586205 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.586468 (+ 263us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.586485 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.586514 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.587043 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.587049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.587052 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.587123 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.587161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260620 09:05:35.590096 5277 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c751a6e96443f8814278df4d313ba (DEFAULT_TABLE table=test-table [id=211663a5449a43bbad6c86cf975de004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:35.590219 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c751a6e96443f8814278df4d313ba (DEFAULT_TABLE table=test-table [id=211663a5449a43bbad6c86cf975de004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:35.590099 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c751a6e96443f8814278df4d313ba (DEFAULT_TABLE table=test-table [id=211663a5449a43bbad6c86cf975de004]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:35.591091 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c751a6e96443f8814278df4d313ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:35.591110 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c751a6e96443f8814278df4d313ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:35.591110 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c751a6e96443f8814278df4d313ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:35.597096 5491 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Bootstrap starting.
I20260620 09:05:35.598335 5492 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Bootstrap starting.
I20260620 09:05:35.599170 5493 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Bootstrap starting.
I20260620 09:05:35.599375 5491 tablet_bootstrap.cc:654] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:35.600385 5492 tablet_bootstrap.cc:654] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:35.600360 5491 log.cc:824] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:35.601025 5493 tablet_bootstrap.cc:654] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:35.601235 5492 log.cc:824] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:35.601905 5493 log.cc:824] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:35.602784 5491 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: No bootstrap required, opened a new log
I20260620 09:05:35.602975 5491 ts_tablet_manager.cc:1397] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:05:35.603168 5492 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: No bootstrap required, opened a new log
I20260620 09:05:35.603341 5492 ts_tablet_manager.cc:1397] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 09:05:35.603850 5493 tablet_bootstrap.cc:492] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: No bootstrap required, opened a new log
I20260620 09:05:35.604053 5493 ts_tablet_manager.cc:1397] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260620 09:05:35.607580 5493 raft_consensus.cc:348] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.607596 5492 raft_consensus.cc:348] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.607591 5491 raft_consensus.cc:348] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.607822 5493 raft_consensus.cc:374] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:35.607823 5492 raft_consensus.cc:374] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:35.607823 5491 raft_consensus.cc:374] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:35.607877 5493 raft_consensus.cc:729] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f956e6d5f5541478ca5dd52aa20a6c2, State: Initialized, Role: FOLLOWER
I20260620 09:05:35.607877 5491 raft_consensus.cc:729] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23dbf9b2109d44de886d8921a891540c, State: Initialized, Role: FOLLOWER
I20260620 09:05:35.607877 5492 raft_consensus.cc:729] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31fdbfdf46749b2bb3242eed5914bc8, State: Initialized, Role: FOLLOWER
I20260620 09:05:35.608312 5493 consensus_queue.cc:260] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.608327 5491 consensus_queue.cc:260] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.608328 5492 consensus_queue.cc:260] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.608961 5343 heartbeater.cc:503] Master 127.0.98.62:38993 was elected leader, sending a full tablet report...
I20260620 09:05:35.609339 5491 ts_tablet_manager.cc:1428] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:35.609355 5493 ts_tablet_manager.cc:1428] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 09:05:35.609634 5474 heartbeater.cc:503] Master 127.0.98.62:38993 was elected leader, sending a full tablet report...
I20260620 09:05:35.610807 5212 heartbeater.cc:503] Master 127.0.98.62:38993 was elected leader, sending a full tablet report...
I20260620 09:05:35.612100 5492 ts_tablet_manager.cc:1428] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
W20260620 09:05:35.616925 5344 tablet.cc:2369] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:05:35.702934 5213 tablet.cc:2369] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:35.722854 5498 raft_consensus.cc:482] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:35.723112 5498 raft_consensus.cc:504] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.724377 5498 leader_election.cc:288] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c31fdbfdf46749b2bb3242eed5914bc8 (127.0.98.2:44741), 9f956e6d5f5541478ca5dd52aa20a6c2 (127.0.98.1:37733)
I20260620 09:05:35.726266 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.724749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60173 (local address 127.0.98.1:37733)
0620 09:05:35.724864 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.724868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.724883 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:05:35.724976 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.724979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.725022 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.725160 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.725874 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.725877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.725880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.725902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.725924 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.725938 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.726034 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.726098 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.726137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:35.726266 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.724654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37733 (local address 127.0.98.3:60173)
0620 09:05:35.724779 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.724795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.724881 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.725181 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.725184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.725199 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.725719 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.725720 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.725730 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.725755 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.726020 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.726023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.726026 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.726103 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.726137 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 09:05:35.726810 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.724660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44741 (local address 127.0.98.3:46619)
0620 09:05:35.725124 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.725135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:35.725248 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:35.725659 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:35.725662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:35.725678 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:35.726194 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:05:35.726196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:35.726208 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:05:35.726235 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:35.726552 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:35.726555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:35.726559 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:05:35.726640 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:05:35.726675 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":126,"thread_start_us":128,"threads_started":1}
I20260620 09:05:35.726819 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.724758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46619 (local address 127.0.98.2:44741)
0620 09:05:35.724890 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.724897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.725224 (+ 327us) server_negotiation.cc:407] Connection header received
0620 09:05:35.725359 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.725362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.725399 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.725657 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.726367 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.726372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.726374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.726390 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.726413 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.726423 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.726505 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.726654 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.726692 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:35.726960 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c751a6e96443f8814278df4d313ba" candidate_uuid: "23dbf9b2109d44de886d8921a891540c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" is_pre_election: true
I20260620 09:05:35.727290 5166 raft_consensus.cc:2454] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23dbf9b2109d44de886d8921a891540c in term 0.
I20260620 09:05:35.727427 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c751a6e96443f8814278df4d313ba" candidate_uuid: "23dbf9b2109d44de886d8921a891540c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" is_pre_election: true
I20260620 09:05:35.727735 5297 raft_consensus.cc:2454] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23dbf9b2109d44de886d8921a891540c in term 0.
I20260620 09:05:35.727816 5361 leader_election.cc:302] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dbf9b2109d44de886d8921a891540c, 9f956e6d5f5541478ca5dd52aa20a6c2; no voters:
I20260620 09:05:35.728119 5498 raft_consensus.cc:2781] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:35.728205 5498 raft_consensus.cc:482] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:35.728257 5498 raft_consensus.cc:3037] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:05:35.800113 5475 tablet.cc:2369] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:35.816159 5498 raft_consensus.cc:504] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.816704 5498 leader_election.cc:288] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [CANDIDATE]: Term 1 election: Requested vote from peers c31fdbfdf46749b2bb3242eed5914bc8 (127.0.98.2:44741), 9f956e6d5f5541478ca5dd52aa20a6c2 (127.0.98.1:37733)
I20260620 09:05:35.817148 5297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c751a6e96443f8814278df4d313ba" candidate_uuid: "23dbf9b2109d44de886d8921a891540c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31fdbfdf46749b2bb3242eed5914bc8"
I20260620 09:05:35.817370 5297 raft_consensus.cc:3037] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:35.817432 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c751a6e96443f8814278df4d313ba" candidate_uuid: "23dbf9b2109d44de886d8921a891540c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2"
I20260620 09:05:35.817603 5166 raft_consensus.cc:3037] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:35.818663 5297 raft_consensus.cc:2454] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23dbf9b2109d44de886d8921a891540c in term 1.
I20260620 09:05:35.819005 5362 leader_election.cc:302] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23dbf9b2109d44de886d8921a891540c, c31fdbfdf46749b2bb3242eed5914bc8; no voters:
I20260620 09:05:35.819216 5498 raft_consensus.cc:2781] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:35.819226 5166 raft_consensus.cc:2454] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23dbf9b2109d44de886d8921a891540c in term 1.
I20260620 09:05:35.819525 5498 raft_consensus.cc:686] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 1 LEADER]: Becoming Leader. State: Replica: 23dbf9b2109d44de886d8921a891540c, State: Running, Role: LEADER
I20260620 09:05:35.819945 5498 consensus_queue.cc:237] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } }
I20260620 09:05:35.823196 5022 catalog_manager.cc:5495] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c reported cstate change: term changed from 0 to 1, leader changed from <none> to 23dbf9b2109d44de886d8921a891540c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "23dbf9b2109d44de886d8921a891540c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23dbf9b2109d44de886d8921a891540c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42403 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:35.879760 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.878322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.98.3:42403)
0620 09:05:35.878476 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:35.878479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:35.878493 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:35.878568 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:35.878571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:35.878631 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:35.878704 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:35.879358 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:35.879361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:35.879363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:35.879374 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:35.879394 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:35.879402 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:35.879503 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:35.879614 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 09:05:35.879652 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:05:35.885885 5166 raft_consensus.cc:1261] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2 [term 1 FOLLOWER]: Refusing update from remote peer 23dbf9b2109d44de886d8921a891540c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:35.885898 5297 raft_consensus.cc:1261] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 1 FOLLOWER]: Refusing update from remote peer 23dbf9b2109d44de886d8921a891540c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:35.886684 5501 consensus_queue.cc:1035] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f956e6d5f5541478ca5dd52aa20a6c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:35.886972 5498 consensus_queue.cc:1035] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31fdbfdf46749b2bb3242eed5914bc8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:35.896119 5505 mvcc.cc:205] Tried to move back new op lower bound from 7298852191780835328 to 7298852191521218560. Current Snapshot: MvccSnapshot[applied={T|T < 7298852191780835328}]
I20260620 09:05:35.898715 5506 mvcc.cc:205] Tried to move back new op lower bound from 7298852191780835328 to 7298852191521218560. Current Snapshot: MvccSnapshot[applied={T|T < 7298852191780835328}]
I20260620 09:05:35.901459 5023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:35.901597 5023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:35.903829 5023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 449863d8cc634163bdecbfba2f903272: Sending DeleteTablet for 3 replicas of tablet 2d4c751a6e96443f8814278df4d313ba
I20260620 09:05:35.904549 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c751a6e96443f8814278df4d313ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:35 UTC) from {username='slave'} at 127.0.0.1:41598
I20260620 09:05:35.904733 5277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c751a6e96443f8814278df4d313ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:35 UTC) from {username='slave'} at 127.0.0.1:36816
I20260620 09:05:35.904767 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c751a6e96443f8814278df4d313ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:35 UTC) from {username='slave'} at 127.0.0.1:54792
I20260620 09:05:35.905278 5514 tablet_replica.cc:330] T 2d4c751a6e96443f8814278df4d313ba P 9f956e6d5f5541478ca5dd52aa20a6c2: stopping tablet replica
I20260620 09:05:35.905567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5084
W20260620 09:05:35.911891 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37733 (error 108)
I20260620 09:05:35.912061 5515 tablet_replica.cc:330] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8: stopping tablet replica
I20260620 09:05:35.912257 5516 tablet_replica.cc:330] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: stopping tablet replica
I20260620 09:05:35.912421 5515 raft_consensus.cc:2229] T 2d4c751a6e96443f8814278df4d313ba P c31fdbfdf46749b2bb3242eed5914bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:35.912648 5516 raft_consensus.cc:2229] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:35.912690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5215
I20260620 09:05:35.913121 5516 raft_consensus.cc:2258] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:35.914791 5516 ts_tablet_manager.cc:1905] T 2d4c751a6e96443f8814278df4d313ba P 23dbf9b2109d44de886d8921a891540c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:35.918419 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.918092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37733 (local address 127.0.0.1:41604)
0620 09:05:35.918241 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.918316 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 09:05:35.918684 5008 catalog_manager.cc:4584] TS 9f956e6d5f5541478ca5dd52aa20a6c2 (127.0.98.1:37733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d4c751a6e96443f8814278df4d313ba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37733: connect: Connection refused (error 111)
I20260620 09:05:35.919721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5346
I20260620 09:05:35.919832 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.919589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44741 (local address 127.0.0.1:36818)
0620 09:05:35.919700 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.919747 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:05:35.965498 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.965226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44741 (local address 127.0.0.1:36830)
0620 09:05:35.965370 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.965429 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 09:05:35.966195 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.965985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44741 (local address 127.0.0.1:36838)
0620 09:05:35.966092 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.966145 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:05:35.972018 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.971798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37733 (local address 127.0.0.1:41620)
0620 09:05:35.971903 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.971955 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 09:05:35.972607 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:35.972411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37733 (local address 127.0.0.1:41636)
0620 09:05:35.972509 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:35.972563 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:05:36.009934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 4993
2026-06-20T09:05:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1230 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 09:05:36.031641 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:05:36.033282 392 webserver.cc:460] Webserver started at https://127.0.0.1:35839/ using document root <none> and password file <none>
I20260620 09:05:36.033469 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:05:36.033740 392 webserver.cc:460] Webserver started at http://127.0.0.1:33445/ using document root <none> and password file <none>
2026-06-20T09:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:36Z Disabled control of system clock
I20260620 09:05:36.042510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36553
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36553
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/testchainca.cert with env {}
W20260620 09:05:36.152987 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:36.153317 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:36.153357 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:36.157371 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:36.157421 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:36.157436 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:36.157454 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:36.157469 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:05:36.157485 5523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:05:36.157497 5523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35839/jwks.json
I20260620 09:05:36.162704 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:36.163653 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:36.164723 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:36.172948 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:36.172995 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:36.173080 5523 server_base.cc:1034] running on GCE node
W20260620 09:05:36.172958 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:36.173777 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:36.174836 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:36.175992 5523 hybrid_clock.cc:648] HybridClock initialized: now 1781946336175961 us; error 46 us; skew 500 ppm
I20260620 09:05:36.177841 5523 webserver.cc:460] Webserver started at http://127.0.98.62:42065/ using document root <none> and password file <none>
I20260620 09:05:36.178392 5523 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:36.178452 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:36.178678 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:36.180640 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "b4acd3cfbc8a4d64b85ce5499a311f48"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "9dd880fd46a02e51d44e7377b56664b3"
server_key_iv: "666ee5b7163cc1622091a2a38f68d0af"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.181243 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4acd3cfbc8a4d64b85ce5499a311f48"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "9dd880fd46a02e51d44e7377b56664b3"
server_key_iv: "666ee5b7163cc1622091a2a38f68d0af"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.185004 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:05:36.187436 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:36.188612 5523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:36.188727 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "b4acd3cfbc8a4d64b85ce5499a311f48"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "9dd880fd46a02e51d44e7377b56664b3"
server_key_iv: "666ee5b7163cc1622091a2a38f68d0af"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.188822 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:36.199573 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:36.200171 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:36.200366 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:36.206898 5523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36553
I20260620 09:05:36.206905 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36553 every 8 connection(s)
I20260620 09:05:36.207791 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:36.210870 5590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:36.216180 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: Bootstrap starting.
I20260620 09:05:36.217621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5523
I20260620 09:05:36.217751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:36.218004 392 external_mini_cluster.cc:1428] Setting key b7f2aad76c8a047bfe64595d9f4c4e99
I20260620 09:05:36.218786 5590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:36.219496 5590 log.cc:824] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:36.221529 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: No bootstrap required, opened a new log
I20260620 09:05:36.225603 5590 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260620 09:05:36.225867 5590 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:36.225926 5590 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4acd3cfbc8a4d64b85ce5499a311f48, State: Initialized, Role: FOLLOWER
I20260620 09:05:36.226385 5590 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260620 09:05:36.226563 5590 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:36.226648 5590 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:36.226770 5590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:36.227079 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.219031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.0.98.62:36553)
0620 09:05:36.219689 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.219705 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.219740 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:36.220505 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.220524 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.220834 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.221171 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:36.221503 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.222462 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.223721 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.224157 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.224266 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.225170 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:36.225175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:36.225186 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:36.225560 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:36.225634 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:36.225653 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:36.225725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:36.225928 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:05:36.226213 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":142,"threads_started":1}
I20260620 09:05:36.227779 5590 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260620 09:05:36.228117 5590 leader_election.cc:302] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4acd3cfbc8a4d64b85ce5499a311f48; no voters:
I20260620 09:05:36.228428 5590 leader_election.cc:288] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:36.228498 5595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:36.228685 5595 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [term 1 LEADER]: Becoming Leader. State: Replica: b4acd3cfbc8a4d64b85ce5499a311f48, State: Running, Role: LEADER
I20260620 09:05:36.228988 5595 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260620 09:05:36.229707 5590 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:36.229962 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:05:36.230156 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4acd3cfbc8a4d64b85ce5499a311f48. Latest consensus state: current_term: 1 leader_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } } }
I20260620 09:05:36.230269 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:36.232193 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4acd3cfbc8a4d64b85ce5499a311f48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } } }
I20260620 09:05:36.232326 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48 [sys.catalog]: This master's current role is: LEADER
W20260620 09:05:36.234477 5611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:36.234539 5611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:36.234616 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:36.235275 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:36.240589 5599 catalog_manager.cc:1347] Generated new cluster ID: 984419a41e0f44359f3ef67745fb51bc
I20260620 09:05:36.240658 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:36.248492 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:36.249449 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:36.260502 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4acd3cfbc8a4d64b85ce5499a311f48: Generated new TSK 0
I20260620 09:05:36.261206 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:05:36.293121 5620 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 09:05:36.293258 5620 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:05:36.293735 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.271880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.0.98.62:36553)
0620 09:05:36.271997 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.272003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.272199 (+ 196us) server_negotiation.cc:407] Connection header received
0620 09:05:36.272423 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.272428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.272479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.272586 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:05:36.272842 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.273428 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.274330 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.274557 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.274668 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.274985 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:05:36.293381 (+ 18396us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:05:36.293620 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 09:05:36.293830 5593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 09:05:36.295208 5619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:36.271798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36553 (local address 127.0.0.1:34698)
0620 09:05:36.272119 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:36.272131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:36.272268 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:36.272604 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:36.272607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:36.272615 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:05:36.272736 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.272742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.273554 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.273557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.274195 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.274202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.274657 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.274673 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.274761 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.274811 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:05:36.293629 (+ 18818us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 09:05:36.293892 (+ 263us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36553: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36553: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 09:05:36.295921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5523
2026-06-20T09:05:36Z chronyd exiting
I20260620 09:05:36.333617 392 test_util.cc:175] -----------------------------------------------
I20260620 09:05:36.333722 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781946309842113-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 09:05:36.334789 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:05:36.335634 392 webserver.cc:460] Webserver started at https://127.0.0.1:45097/ using document root <none> and password file <none>
I20260620 09:05:36.335759 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:05:36.336052 392 webserver.cc:460] Webserver started at http://127.0.0.1:39985/ using document root <none> and password file <none>
2026-06-20T09:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:36Z Disabled control of system clock
I20260620 09:05:36.344471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36307
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/testchainca.cert with env {}
W20260620 09:05:36.452149 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:36.452399 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:36.452440 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:36.456398 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:36.456457 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:36.456472 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:36.456489 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:36.456504 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:05:36.456521 5626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:05:36.456544 5626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45097/jwks.json
I20260620 09:05:36.461766 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:36.462734 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:36.463646 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:36.471619 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:36.471728 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:36.471885 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:36.471743 5626 server_base.cc:1034] running on GCE node
I20260620 09:05:36.472460 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:36.473577 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:36.474756 5626 hybrid_clock.cc:648] HybridClock initialized: now 1781946336474750 us; error 31 us; skew 500 ppm
I20260620 09:05:36.476450 5626 webserver.cc:460] Webserver started at http://127.0.98.62:38975/ using document root <none> and password file <none>
I20260620 09:05:36.477016 5626 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:36.477157 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:36.477375 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:36.479411 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "3cc3686744880947ab084d49a293fe81"
server_key_iv: "a3a8581643224c21f820b4f0c73a82a0"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.479969 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "3cc3686744880947ab084d49a293fe81"
server_key_iv: "a3a8581643224c21f820b4f0c73a82a0"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.483516 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:05:36.485891 5640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:36.487072 5626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:05:36.487203 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "3cc3686744880947ab084d49a293fe81"
server_key_iv: "a3a8581643224c21f820b4f0c73a82a0"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.487313 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:36.502090 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:36.502771 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:36.502964 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:36.510576 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36307
I20260620 09:05:36.510591 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36307 every 8 connection(s)
I20260620 09:05:36.511562 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:36.514652 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:36.519802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5626
I20260620 09:05:36.519986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:36.520006 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd: Bootstrap starting.
I20260620 09:05:36.520275 392 external_mini_cluster.cc:1428] Setting key 16e9424d6ea2236d8122676388b9d4ab
I20260620 09:05:36.522657 5693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:36.523500 5693 log.cc:824] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:36.525699 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd: No bootstrap required, opened a new log
I20260620 09:05:36.530144 5693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260620 09:05:36.530392 5693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:36.530426 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.521546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.0.98.62:36307)
0620 09:05:36.522191 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.522209 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.522251 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:05:36.523132 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.523154 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.523645 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.523998 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:36.524386 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.525812 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.526660 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.527374 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.527561 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.528304 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:36.528310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:36.528325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:36.528822 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:36.528893 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:36.528924 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:36.529069 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:36.529216 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 09:05:36.529508 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260620 09:05:36.530454 5693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed9c59dea2b4ed484f7fb0bff3491cd, State: Initialized, Role: FOLLOWER
I20260620 09:05:36.530900 5693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260620 09:05:36.531023 5693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:36.531107 5693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:36.531231 5693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:36.532312 5693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260620 09:05:36.532727 5693 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed9c59dea2b4ed484f7fb0bff3491cd; no voters:
I20260620 09:05:36.533073 5693 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:36.533180 5698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:36.533350 5698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [term 1 LEADER]: Becoming Leader. State: Replica: 6ed9c59dea2b4ed484f7fb0bff3491cd, State: Running, Role: LEADER
I20260620 09:05:36.533452 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:05:36.533635 5698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260620 09:05:36.534345 5693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:36.535039 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260620 09:05:36.535169 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:36.535524 5705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:36.535490 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed9c59dea2b4ed484f7fb0bff3491cd. Latest consensus state: current_term: 1 leader_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9c59dea2b4ed484f7fb0bff3491cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260620 09:05:36.535584 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:36.538581 5705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:36.545560 5705 catalog_manager.cc:1347] Generated new cluster ID: 0e36e0eb49b3497289445c3b315ace38
I20260620 09:05:36.545673 5705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:36.559381 5705 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:36.560581 5705 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:36.575608 5705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed9c59dea2b4ed484f7fb0bff3491cd: Generated new TSK 0
I20260620 09:05:36.576447 5705 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 09:05:36.599788 5723 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 09:05:36.599877 5723 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 09:05:36.600489 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.584929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.98.62:36307)
0620 09:05:36.585073 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.585078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.585275 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:05:36.585426 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.585428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.585479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.585568 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:05:36.585825 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.586335 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.587548 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.587753 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.587824 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.588110 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:05:36.600163 (+ 12053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:05:36.600373 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:05:36.600584 5696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 09:05:36.600739 5722 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:36.584873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.0.1:52154)
0620 09:05:36.585185 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:36.585200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:36.585317 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:36.585593 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:36.585596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:36.585604 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:05:36.585723 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.585729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.586433 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.586436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.587443 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.587453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.587840 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.587858 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.587946 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.587973 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:05:36.600356 (+ 12383us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 09:05:36.600558 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36307: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260620 09:05:36.601310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5626
2026-06-20T09:05:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 09:05:36.640316 392 mini_oidc.cc:152] Starting JWKS server
I20260620 09:05:36.641217 392 webserver.cc:460] Webserver started at https://127.0.0.1:41421/ using document root <none> and password file <none>
I20260620 09:05:36.641299 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 09:05:36.641497 392 webserver.cc:460] Webserver started at http://127.0.0.1:44603/ using document root <none> and password file <none>
2026-06-20T09:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:36Z Disabled control of system clock
I20260620 09:05:36.650147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43711
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:05:36.757995 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:36.758257 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:36.758294 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:36.762110 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:36.762162 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:36.762177 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:36.762193 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:36.762208 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:05:36.762223 5729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:05:36.762235 5729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41421/jwks.json
I20260620 09:05:36.767060 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:36.768069 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:36.769393 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:36.777546 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:36.777557 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:36.777570 5729 server_base.cc:1034] running on GCE node
W20260620 09:05:36.777588 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:36.778278 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:36.779340 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:36.780537 5729 hybrid_clock.cc:648] HybridClock initialized: now 1781946336780519 us; error 27 us; skew 500 ppm
I20260620 09:05:36.786998 5729 webserver.cc:460] Webserver started at http://127.0.98.62:33051/ using document root <none> and password file <none>
I20260620 09:05:36.787599 5729 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:36.787689 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:36.787909 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:36.790014 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "a9f4a9e5ffe748bda7241f0be9d0b787"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "2814133d4b6789052c387695ff3ceffa"
server_key_iv: "596ec4fdd6a818769307462055d8ffdd"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.790567 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9f4a9e5ffe748bda7241f0be9d0b787"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "2814133d4b6789052c387695ff3ceffa"
server_key_iv: "596ec4fdd6a818769307462055d8ffdd"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.794216 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:36.796480 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:36.797742 5729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:36.797870 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "a9f4a9e5ffe748bda7241f0be9d0b787"
format_stamp: "Formatted at 2026-06-20 09:05:36 on dist-test-slave-92t4"
server_key: "2814133d4b6789052c387695ff3ceffa"
server_key_iv: "596ec4fdd6a818769307462055d8ffdd"
server_key_version: "encryptionkey@0"
I20260620 09:05:36.797983 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:36.822154 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:36.822831 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:36.823024 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:36.829521 5729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43711
I20260620 09:05:36.829550 5795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43711 every 8 connection(s)
I20260620 09:05:36.830420 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:36.833515 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:36.835695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5729
I20260620 09:05:36.835817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:36.836105 392 external_mini_cluster.cc:1428] Setting key 023e3917614da32f06125cbfd516c5d0
I20260620 09:05:36.839557 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787: Bootstrap starting.
I20260620 09:05:36.841816 5796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:36.842517 5796 log.cc:824] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:36.844517 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787: No bootstrap required, opened a new log
I20260620 09:05:36.845155 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.837505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.0.98.62:43711)
0620 09:05:36.838105 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.838119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.838151 (+ 32us) server_negotiation.cc:407] Connection header received
0620 09:05:36.838772 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.838788 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.839165 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.839497 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:36.839899 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.840845 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.841802 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.842375 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.842544 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.843389 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:36.843395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:36.843406 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:36.843774 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:36.843827 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:36.843845 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:36.843919 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:36.844081 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 09:05:36.844360 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260620 09:05:36.847672 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:05:36.848145 5796 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } }
I20260620 09:05:36.848408 5796 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:36.848465 5796 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f4a9e5ffe748bda7241f0be9d0b787, State: Initialized, Role: FOLLOWER
I20260620 09:05:36.848896 5796 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } }
I20260620 09:05:36.849007 5796 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:36.849114 5796 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:36.849229 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:36.850080 5796 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } }
I20260620 09:05:36.850399 5796 leader_election.cc:302] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f4a9e5ffe748bda7241f0be9d0b787; no voters:
I20260620 09:05:36.850692 5796 leader_election.cc:288] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:36.850798 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:36.851038 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [term 1 LEADER]: Becoming Leader. State: Replica: a9f4a9e5ffe748bda7241f0be9d0b787, State: Running, Role: LEADER
I20260620 09:05:36.851321 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } }
I20260620 09:05:36.851795 5796 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:36.852944 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } } }
I20260620 09:05:36.852944 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f4a9e5ffe748bda7241f0be9d0b787. Latest consensus state: current_term: 1 leader_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f4a9e5ffe748bda7241f0be9d0b787" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43711 } } }
I20260620 09:05:36.853112 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:36.853112 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:36.853549 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:36.856539 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:36.861904 5810 catalog_manager.cc:1347] Generated new cluster ID: 1ceff4ddce034a8b8d364907cc6036db
I20260620 09:05:36.861965 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:36.869014 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:36.869894 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:36.873648 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f4a9e5ffe748bda7241f0be9d0b787: Generated new TSK 0
I20260620 09:05:36.874228 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:05:36.889715 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:36.884411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.0.98.62:43711)
0620 09:05:36.884555 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:36.884559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:36.884738 (+ 179us) server_negotiation.cc:407] Connection header received
0620 09:05:36.884890 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:36.884893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:36.884935 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:36.885018 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:05:36.885275 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.886178 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.887428 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.887641 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.887711 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.888020 (+ 309us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:05:36.889338 (+ 1318us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41421/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:05:36.889563 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57476: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41421/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 09:05:36.889778 5799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57476: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41421/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 09:05:36.891585 5825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:36.884327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43711 (local address 127.0.0.1:57476)
0620 09:05:36.884646 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:36.884661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:36.884780 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:36.885057 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:36.885060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:36.885068 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:05:36.885176 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.885183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.886309 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.886314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.887314 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:36.887324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:36.887729 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:36.887752 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:36.887842 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:36.887870 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:05:36.889498 (+ 1628us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41421/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 09:05:36.891399 (+ 1901us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41421/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1652,"thread_start_us":114,"threads_started":1}
I20260620 09:05:36.892144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5729
E20260620 09:05:36.900825 5826 webserver.cc:533] Webserver: sslize failed:
E20260620 09:05:36.900885 5826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T09:05:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T09:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:36Z Disabled control of system clock
I20260620 09:05:36.939428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33207
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33207
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/testchainca.cert with env {}
W20260620 09:05:37.046881 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:37.047159 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:37.047199 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:37.051107 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:37.051162 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:37.051177 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:37.051194 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:37.051209 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 09:05:37.051225 5830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 09:05:37.051239 5830 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 09:05:37.056428 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:37.057461 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:37.058398 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:37.066430 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:37.066437 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:37.066661 5830 server_base.cc:1034] running on GCE node
W20260620 09:05:37.066489 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:37.067399 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:37.068486 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:37.069682 5830 hybrid_clock.cc:648] HybridClock initialized: now 1781946337069650 us; error 44 us; skew 500 ppm
I20260620 09:05:37.071419 5830 webserver.cc:460] Webserver started at http://127.0.98.62:39827/ using document root <none> and password file <none>
I20260620 09:05:37.071992 5830 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:37.072084 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:37.072305 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:37.074468 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "145b524812014da5a043c8150a93e7f0"
format_stamp: "Formatted at 2026-06-20 09:05:37 on dist-test-slave-92t4"
server_key: "527b1f5cd77f66554ee43053f79ffe1d"
server_key_iv: "ba940a292fe47f3d84de970f2decc64d"
server_key_version: "encryptionkey@0"
I20260620 09:05:37.075030 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "145b524812014da5a043c8150a93e7f0"
format_stamp: "Formatted at 2026-06-20 09:05:37 on dist-test-slave-92t4"
server_key: "527b1f5cd77f66554ee43053f79ffe1d"
server_key_iv: "ba940a292fe47f3d84de970f2decc64d"
server_key_version: "encryptionkey@0"
I20260620 09:05:37.078608 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:05:37.080935 5844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:37.082134 5830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:37.082271 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "145b524812014da5a043c8150a93e7f0"
format_stamp: "Formatted at 2026-06-20 09:05:37 on dist-test-slave-92t4"
server_key: "527b1f5cd77f66554ee43053f79ffe1d"
server_key_iv: "ba940a292fe47f3d84de970f2decc64d"
server_key_version: "encryptionkey@0"
I20260620 09:05:37.082382 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:37.101543 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:37.102226 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:37.102419 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:37.109265 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33207
I20260620 09:05:37.109303 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33207 every 8 connection(s)
I20260620 09:05:37.110224 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:37.113273 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:37.114773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5830
I20260620 09:05:37.114908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:37.115171 392 external_mini_cluster.cc:1428] Setting key 78513576fd554c7f64ce1a79ddb5d437
I20260620 09:05:37.119827 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0: Bootstrap starting.
I20260620 09:05:37.122386 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:37.123106 5897 log.cc:824] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:37.125217 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:37.116506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.98.62:33207)
0620 09:05:37.117197 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:37.117216 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:37.117257 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:05:37.118046 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:37.118068 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:37.118542 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:37.118928 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:37.119350 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.120702 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.121588 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.122250 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.122413 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:37.123252 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:37.123258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:37.123273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:05:37.123726 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:37.123812 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:05:37.123839 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:37.123933 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:37.124073 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 09:05:37.124321 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":98,"threads_started":1}
I20260620 09:05:37.125243 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0: No bootstrap required, opened a new log
I20260620 09:05:37.127873 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:05:37.129246 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260620 09:05:37.129546 5897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:37.129606 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145b524812014da5a043c8150a93e7f0, State: Initialized, Role: FOLLOWER
I20260620 09:05:37.130055 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260620 09:05:37.130203 5897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:37.130270 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:37.130365 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:37.131222 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260620 09:05:37.131558 5897 leader_election.cc:302] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 145b524812014da5a043c8150a93e7f0; no voters:
I20260620 09:05:37.131840 5897 leader_election.cc:288] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:37.131943 5902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:37.132189 5902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [term 1 LEADER]: Becoming Leader. State: Replica: 145b524812014da5a043c8150a93e7f0, State: Running, Role: LEADER
I20260620 09:05:37.132493 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } }
I20260620 09:05:37.133129 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:37.134315 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "145b524812014da5a043c8150a93e7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } } }
I20260620 09:05:37.134327 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 145b524812014da5a043c8150a93e7f0. Latest consensus state: current_term: 1 leader_uuid: "145b524812014da5a043c8150a93e7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145b524812014da5a043c8150a93e7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33207 } } }
I20260620 09:05:37.134469 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:37.134473 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:37.134832 5911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:37.137821 5911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:37.143824 5911 catalog_manager.cc:1347] Generated new cluster ID: 4af060888f4747b69dce2bd5467012d0
I20260620 09:05:37.143890 5911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:37.156343 5911 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:37.157541 5911 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:37.164875 5911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 145b524812014da5a043c8150a93e7f0: Generated new TSK 0
I20260620 09:05:37.165680 5911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:05:37.205504 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:37.181147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.0.98.62:33207)
0620 09:05:37.181283 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:37.181288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:37.181474 (+ 186us) server_negotiation.cc:407] Connection header received
0620 09:05:37.181616 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:37.181618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:37.181662 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:37.181749 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0620 09:05:37.182012 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.182530 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.183704 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.183913 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.183980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:37.184329 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 09:05:37.205108 (+ 20779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:05:37.205365 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:05:37.205615 5900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 09:05:37.205711 5926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:37.181074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33207 (local address 127.0.0.1:42716)
0620 09:05:37.181380 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:37.181396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:37.181512 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:37.181776 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:37.181779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:37.181787 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 09:05:37.181908 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:37.181915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.182640 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.182643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:37.183592 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:37.183601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:37.184009 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:37.184037 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:37.184146 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:37.184173 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 09:05:37.205325 (+ 21152us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 09:05:37.205524 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":119,"threads_started":1}
I20260620 09:05:37.206502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5830
2026-06-20T09:05:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (293 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:37 dist-test-slave-92t4 krb5kdc[5938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:37 dist-test-slave-92t4 krb5kdc[5938](info): set up 2 sockets
Jun 20 09:05:37 dist-test-slave-92t4 krb5kdc[5938](info): commencing operation
krb5kdc: starting...
W20260620 09:05:38.473312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:38 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:38.589376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:38.696592 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:38.696834 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:38.696872 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:38.700666 5951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:38.700711 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:38.700729 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:38.700745 5951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:38.700759 5951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:38.705634 5951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:38.706542 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:38.707470 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:38.715523 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:38.715535 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:38.715530 5959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:38.715597 5951 server_base.cc:1034] running on GCE node
I20260620 09:05:38.716305 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:38.717423 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:38.718585 5951 hybrid_clock.cc:648] HybridClock initialized: now 1781946338718574 us; error 23 us; skew 500 ppm
Jun 20 09:05:38 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:38.721202 5951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:38.721997 5951 webserver.cc:460] Webserver started at http://127.0.98.62:42273/ using document root <none> and password file <none>
I20260620 09:05:38.722572 5951 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:38.722659 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:38.722879 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:38.724887 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "c539c8fdbb87410c971b7659afb64904"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "e40676a21d138a21272a2b85ca9d6546"
server_key_iv: "d9b4f4beeeb790bdd57a8300fb918d07"
server_key_version: "encryptionkey@0"
I20260620 09:05:38.725515 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "c539c8fdbb87410c971b7659afb64904"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "e40676a21d138a21272a2b85ca9d6546"
server_key_iv: "d9b4f4beeeb790bdd57a8300fb918d07"
server_key_version: "encryptionkey@0"
I20260620 09:05:38.728966 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:38.731218 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:38.732331 5951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:38.732487 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "c539c8fdbb87410c971b7659afb64904"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "e40676a21d138a21272a2b85ca9d6546"
server_key_iv: "d9b4f4beeeb790bdd57a8300fb918d07"
server_key_version: "encryptionkey@0"
I20260620 09:05:38.732597 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:38.748116 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:38.750847 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:38.751057 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:38.757531 5951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35143
I20260620 09:05:38.757815 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35143 every 8 connection(s)
I20260620 09:05:38.758539 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:38.761317 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:38.764871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 5951
I20260620 09:05:38.765014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:38.765308 392 external_mini_cluster.cc:1428] Setting key ce2c5c883739a00b0d0001afe0b74f6c
I20260620 09:05:38.766911 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904: Bootstrap starting.
I20260620 09:05:38.769107 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:38.770004 6019 log.cc:824] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:38.771938 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904: No bootstrap required, opened a new log
Jun 20 09:05:38 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:38.775904 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260620 09:05:38.776144 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:38.776216 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c539c8fdbb87410c971b7659afb64904, State: Initialized, Role: FOLLOWER
I20260620 09:05:38.776669 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260620 09:05:38.776829 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:38.776908 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:38.777004 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:38.778191 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260620 09:05:38.778535 6019 leader_election.cc:302] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c539c8fdbb87410c971b7659afb64904; no voters:
I20260620 09:05:38.778861 6019 leader_election.cc:288] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:38.778954 6024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:38.779215 6024 raft_consensus.cc:686] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [term 1 LEADER]: Becoming Leader. State: Replica: c539c8fdbb87410c971b7659afb64904, State: Running, Role: LEADER
I20260620 09:05:38.779567 6024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } }
I20260620 09:05:38.780123 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:38.780251 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:38.766642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.98.62:35143)
0620 09:05:38.767114 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:38.767130 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:38.767165 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:38.767784 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:38.767800 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:38.768116 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:38.768443 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:38.769309 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:38.770675 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:38.771542 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:38.772209 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:38.772369 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:38.775145 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:38.775150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:38.775162 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:38.775186 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:38.777165 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:38.777624 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:38.777629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:38.777633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:38.777732 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:38.778047 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:38.778049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:38.778050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:38.778468 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:38.778604 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:38.778972 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 09:05:38.779290 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260620 09:05:38.781204 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c539c8fdbb87410c971b7659afb64904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } } }
I20260620 09:05:38.781311 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:38.781672 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c539c8fdbb87410c971b7659afb64904. Latest consensus state: current_term: 1 leader_uuid: "c539c8fdbb87410c971b7659afb64904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c539c8fdbb87410c971b7659afb64904" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35143 } } }
I20260620 09:05:38.781802 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:38.782739 6033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:38.785204 6033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:38.790745 6033 catalog_manager.cc:1347] Generated new cluster ID: 6ad07649c26a4466ba1b2e3dc090b300
I20260620 09:05:38.790813 6033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:38.813586 6033 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:38.814605 6033 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:38.822683 6033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c539c8fdbb87410c971b7659afb64904: Generated new TSK 0
I20260620 09:05:38.823462 6033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:38.857971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:38.965477 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:38.965727 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:38.965770 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:38.969862 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:38.969923 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:38.969978 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:38.975000 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:38.975948 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:38.976903 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:38.986223 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:38.986236 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:38.986209 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:38.986536 6045 server_base.cc:1034] running on GCE node
I20260620 09:05:38.986948 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:38.987555 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:38.988741 6045 hybrid_clock.cc:648] HybridClock initialized: now 1781946338988724 us; error 26 us; skew 500 ppm
Jun 20 09:05:38 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:38.991452 6045 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:38.992244 6045 webserver.cc:460] Webserver started at http://127.0.98.1:44135/ using document root <none> and password file <none>
I20260620 09:05:38.992827 6045 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:38.992915 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:38.993211 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:38.995234 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "a626cd462825445aa7bd8054b7d72159"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "03922d3284c76a23491658ee0d3cdbdf"
server_key_iv: "f827398bdf4883a2896bd21bccf377f2"
server_key_version: "encryptionkey@0"
I20260620 09:05:38.995795 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a626cd462825445aa7bd8054b7d72159"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "03922d3284c76a23491658ee0d3cdbdf"
server_key_iv: "f827398bdf4883a2896bd21bccf377f2"
server_key_version: "encryptionkey@0"
I20260620 09:05:38.999558 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:05:39.001909 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.003217 6045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:05:39.003345 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "a626cd462825445aa7bd8054b7d72159"
format_stamp: "Formatted at 2026-06-20 09:05:38 on dist-test-slave-92t4"
server_key: "03922d3284c76a23491658ee0d3cdbdf"
server_key_iv: "f827398bdf4883a2896bd21bccf377f2"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.003485 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:39.015637 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:39.018469 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:39.018687 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:39.019264 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:39.020085 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:39.020174 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.020247 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:39.020293 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.029356 6045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37903
I20260620 09:05:39.029386 6173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37903 every 8 connection(s)
I20260620 09:05:39.030232 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:39.033844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6045
I20260620 09:05:39.033960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:39.034263 392 external_mini_cluster.cc:1428] Setting key 29b80718aeed4009633c72c42716f1f5
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:39.044948 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.031951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38657 (local address 127.0.98.62:35143)
0620 09:05:39.032081 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.032085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.032877 (+ 792us) server_negotiation.cc:407] Connection header received
0620 09:05:39.034585 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.034588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.034659 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.034774 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.036204 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.036788 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.038163 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.038571 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.038704 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.041133 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.041139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.041146 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.041181 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.042928 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.043415 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.043420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.043422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.043522 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.043860 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.043864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.043865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.044121 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.044232 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.044702 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.044812 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:39.045926 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.032333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35143 (local address 127.0.98.1:38657)
0620 09:05:39.032744 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:39.032782 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:39.034338 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:39.034971 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:39.034981 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:39.035336 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:39.036051 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.036072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.036906 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.036913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.038029 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.038037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.038723 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.038726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.038861 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.039468 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:39.039484 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:39.040951 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:39.043037 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.043042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.043053 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.043296 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.043625 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.043627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.043628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.043757 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.044231 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:39.044236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:39.044473 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 09:05:39.044621 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 09:05:39.044867 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:05:39.047155 6174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35143
I20260620 09:05:39.047399 6174 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:39.047883 6174 heartbeater.cc:511] Master 127.0.98.62:35143 requested a full tablet report, sending...
I20260620 09:05:39.049527 5983 ts_manager.cc:194] Registered new tserver with Master: a626cd462825445aa7bd8054b7d72159 (127.0.98.1:37903)
I20260620 09:05:39.051194 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:39.061627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:39.168737 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:39.168972 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:39.169075 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:39.172964 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:39.173022 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:39.173144 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:39.178117 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:39.179105 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:39.180076 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:05:39.189795 6179 server_base.cc:1034] running on GCE node
W20260620 09:05:39.189791 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:39.189795 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:39.189991 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:39.190404 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:39.191063 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:39.192246 6179 hybrid_clock.cc:648] HybridClock initialized: now 1781946339192219 us; error 43 us; skew 500 ppm
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.194986 6179 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:39.195739 6179 webserver.cc:460] Webserver started at http://127.0.98.2:34379/ using document root <none> and password file <none>
I20260620 09:05:39.196259 6179 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:39.196313 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:39.196467 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:39.198472 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "17cbf158bcd3485980cd79ba6c339f9b"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "9dffa971d0aa34b0db836e5b8d6e7a88"
server_key_iv: "bed0ac32983adf69cd1e90bfcdaedef0"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.198968 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "17cbf158bcd3485980cd79ba6c339f9b"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "9dffa971d0aa34b0db836e5b8d6e7a88"
server_key_iv: "bed0ac32983adf69cd1e90bfcdaedef0"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.202451 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:39.204790 6194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.206020 6179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:39.206135 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "17cbf158bcd3485980cd79ba6c339f9b"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "9dffa971d0aa34b0db836e5b8d6e7a88"
server_key_iv: "bed0ac32983adf69cd1e90bfcdaedef0"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.206238 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:39.215456 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:39.218195 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:39.218387 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:39.218916 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:39.219696 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:39.219739 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.219805 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:39.219846 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.228269 6179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44811
I20260620 09:05:39.228291 6307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44811 every 8 connection(s)
I20260620 09:05:39.229168 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:39.237345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6179
I20260620 09:05:39.237507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:39.237795 392 external_mini_cluster.cc:1428] Setting key b7d5835bfa801e9af1a94471a74450a2
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:39.242569 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.230906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43851 (local address 127.0.98.62:35143)
0620 09:05:39.231036 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.231040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.231790 (+ 750us) server_negotiation.cc:407] Connection header received
0620 09:05:39.233177 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.233181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.233236 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.233317 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.234386 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.234925 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.236113 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.236329 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.236444 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.238938 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.238941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.238943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.238987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.240414 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.240969 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.240974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.240977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.241090 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.241407 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.241411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.241413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.241669 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.241781 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.242340 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.242452 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:39.243814 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.231250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35143 (local address 127.0.98.2:43851)
0620 09:05:39.231654 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:39.231695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:39.232910 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:39.233475 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:39.233481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:39.233804 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:39.234247 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.234259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.235054 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.235064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.235912 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.235921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.236476 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.236480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.236667 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.237227 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:39.237242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:39.238788 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:39.240546 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.240553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.240572 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.240872 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.241189 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.241191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.241192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.241319 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.241773 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:39.241777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:39.242097 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 09:05:39.242302 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 09:05:39.242656 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260620 09:05:39.244946 6308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35143
I20260620 09:05:39.245229 6308 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:39.245811 6308 heartbeater.cc:511] Master 127.0.98.62:35143 requested a full tablet report, sending...
I20260620 09:05:39.246876 5983 ts_manager.cc:194] Registered new tserver with Master: 17cbf158bcd3485980cd79ba6c339f9b (127.0.98.2:44811)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:39.247450 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:39.263816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:39.374445 6313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:39.374745 6313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:39.374821 6313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:39.379114 6313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:39.379218 6313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:39.379307 6313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:39.385115 6313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:39.386219 6313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:39.387251 6313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:39.396716 6321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:39.396725 6318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:39.396916 6313 server_base.cc:1034] running on GCE node
W20260620 09:05:39.397058 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:39.397469 6313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:39.398108 6313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:39.399297 6313 hybrid_clock.cc:648] HybridClock initialized: now 1781946339399281 us; error 30 us; skew 500 ppm
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.401952 6313 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:39.402825 6313 webserver.cc:460] Webserver started at http://127.0.98.3:38445/ using document root <none> and password file <none>
I20260620 09:05:39.403369 6313 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:39.403426 6313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:39.403586 6313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:39.405613 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e941818c0fd47a582f5d4bad1cada01"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "87f3fa3519246a2cbd4729d317921c98"
server_key_iv: "8ef14070c5497132b7ecb3b4543939d9"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.406119 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e941818c0fd47a582f5d4bad1cada01"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "87f3fa3519246a2cbd4729d317921c98"
server_key_iv: "8ef14070c5497132b7ecb3b4543939d9"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.409852 6313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:05:39.412145 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.413324 6313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:39.413417 6313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "4e941818c0fd47a582f5d4bad1cada01"
format_stamp: "Formatted at 2026-06-20 09:05:39 on dist-test-slave-92t4"
server_key: "87f3fa3519246a2cbd4729d317921c98"
server_key_iv: "8ef14070c5497132b7ecb3b4543939d9"
server_key_version: "encryptionkey@0"
I20260620 09:05:39.413496 6313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:39.432194 6313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:39.434881 6313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:39.435024 6313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:39.435547 6313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:39.436313 6313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:39.436357 6313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.436394 6313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:39.436416 6313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:39.445083 6313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38719
I20260620 09:05:39.445118 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38719 every 8 connection(s)
I20260620 09:05:39.445932 6313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:39.449550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6313
I20260620 09:05:39.449673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:39.449980 392 external_mini_cluster.cc:1428] Setting key add9d01f330e4006976d03f93db836b2
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:39.458846 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.447618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41505 (local address 127.0.98.62:35143)
0620 09:05:39.447773 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.447778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.448545 (+ 767us) server_negotiation.cc:407] Connection header received
0620 09:05:39.449965 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.449969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.450028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.450125 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.451644 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.452294 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.453380 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.453591 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.453667 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.455941 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.455944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.455946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.455968 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.457192 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.457635 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.457637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.457638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.457706 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.457978 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.457979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.457980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.458137 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.458209 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.458642 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.458730 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:05:39.459604 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.447997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35143 (local address 127.0.98.3:41505)
0620 09:05:39.448403 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:39.448443 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:39.449728 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:39.450315 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:39.450322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:39.450812 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:39.451415 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.451430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.452399 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.452402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.453283 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:39.453291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.453709 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.453712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:39.453844 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.454370 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:39.454384 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:39.455806 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:39.457284 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.457288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.457299 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.457555 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.457781 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:39.457783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:39.457785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:39.457903 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:39.458219 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:39.458223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:39.458447 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 09:05:39.458592 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 09:05:39.458818 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260620 09:05:39.460668 6442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35143
I20260620 09:05:39.460858 6442 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:39.461336 6442 heartbeater.cc:511] Master 127.0.98.62:35143 requested a full tablet report, sending...
I20260620 09:05:39.462298 5983 ts_manager.cc:194] Registered new tserver with Master: 4e941818c0fd47a582f5d4bad1cada01 (127.0.98.3:38719)
I20260620 09:05:39.462925 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41505
I20260620 09:05:39.463727 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/test-admin.keytab.
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.478425 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 09:05:39.481326 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:39.479804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.98.1:37903)
0620 09:05:39.480213 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.480223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.480245 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:05:39.480295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.480300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.480491 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.480608 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.481136 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42474: BlockingRecv error: recv got EOF from 127.0.0.1:42474 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
W20260620 09:05:39.481433 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.482424 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.483317 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:39.492745 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.484185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.98.1:37903)
0620 09:05:39.484307 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.484311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.484403 (+ 92us) server_negotiation.cc:407] Connection header received
0620 09:05:39.484568 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.484575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.484615 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.484700 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.485268 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.486043 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.486889 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.487270 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.487405 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.489618 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.489627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.489639 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.489661 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.491068 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.491484 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.491496 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.491499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.491572 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.491832 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.491836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.491838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.492081 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.492282 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.492455 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.492603 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 09:05:39.496115 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:39.495322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.98.1:37903)
0620 09:05:39.495449 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.495452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.495464 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:39.495560 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.495562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.495593 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.495652 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.495987 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42488: BlockingRecv error: recv got EOF from 127.0.0.1:42488 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:05:39.496189 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.497251 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.498082 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:39.506124 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.498859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.0.98.1:37903)
0620 09:05:39.498977 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.498981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.499081 (+ 100us) server_negotiation.cc:407] Connection header received
0620 09:05:39.499215 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.499218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.499254 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.499337 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.499779 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.500271 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.501020 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.501287 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.501390 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.503414 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.503418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.503420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.503443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.504607 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.505013 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.505019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.505024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.505152 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.505485 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.505487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.505488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.505639 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.505736 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.505923 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.506014 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 09:05:39.508602 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:39.507783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.1:37903)
0620 09:05:39.507913 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.507922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.507940 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:05:39.508043 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.508045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.508078 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.508133 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.508494 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42496: BlockingRecv error: recv got EOF from 127.0.0.1:42496 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:05:39.508674 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781946309842113-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.509671 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:39.510546 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:05:39.517889 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:39.511254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.98.1:37903)
0620 09:05:39.511367 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:39.511370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:39.511405 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:05:39.511500 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:39.511502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:39.511532 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:39.511589 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:39.511966 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.512409 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.513160 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:39.513354 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:39.513426 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:39.515285 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:39.515287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:39.515289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:39.515308 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:39.516465 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.516893 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.516896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.516897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.516972 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:39.517230 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:39.517232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:39.517233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:39.517364 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:39.517453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:39.517610 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 09:05:39.517736 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:39.518965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6045
I20260620 09:05:39.525946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6179
I20260620 09:05:39.531831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6313
I20260620 09:05:39.537032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 5951
2026-06-20T09:05:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2095 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[6462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[6462](info): set up 2 sockets
Jun 20 09:05:39 dist-test-slave-92t4 krb5kdc[6462](info): commencing operation
krb5kdc: starting...
W20260620 09:05:40.569430 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:40 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:05:40.683751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:40.790704 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:40.790937 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:40.790994 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:40.794859 6475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:40.794902 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:40.794922 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:40.794938 6475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:40.794953 6475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:40.799912 6475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:40.800796 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:40.801707 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:40.809773 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:40.809779 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:40.809803 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:40.810046 6475 server_base.cc:1034] running on GCE node
I20260620 09:05:40.810653 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:40.811658 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:40.812840 6475 hybrid_clock.cc:648] HybridClock initialized: now 1781946340812810 us; error 41 us; skew 500 ppm
Jun 20 09:05:40 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:40.815469 6475 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:05:40.816336 6475 webserver.cc:460] Webserver started at http://127.0.98.62:37523/ using document root <none> and password file <none>
I20260620 09:05:40.816831 6475 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:40.816887 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:40.817094 6475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:40.819038 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "7e9a82bd647f4e388487436440d43d76"
format_stamp: "Formatted at 2026-06-20 09:05:40 on dist-test-slave-92t4"
server_key: "cbbb1e249404263ae9970d4992ab9204"
server_key_iv: "d668445de038121f95c884aa99ac7010"
server_key_version: "encryptionkey@0"
I20260620 09:05:40.819535 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e9a82bd647f4e388487436440d43d76"
format_stamp: "Formatted at 2026-06-20 09:05:40 on dist-test-slave-92t4"
server_key: "cbbb1e249404263ae9970d4992ab9204"
server_key_iv: "d668445de038121f95c884aa99ac7010"
server_key_version: "encryptionkey@0"
I20260620 09:05:40.823035 6475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:05:40.825249 6490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:40.826411 6475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:40.826542 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "7e9a82bd647f4e388487436440d43d76"
format_stamp: "Formatted at 2026-06-20 09:05:40 on dist-test-slave-92t4"
server_key: "cbbb1e249404263ae9970d4992ab9204"
server_key_iv: "d668445de038121f95c884aa99ac7010"
server_key_version: "encryptionkey@0"
I20260620 09:05:40.826638 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:40.838461 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:40.841609 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:40.841809 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:40.848140 6475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35565
I20260620 09:05:40.848400 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35565 every 8 connection(s)
I20260620 09:05:40.849152 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:40.849382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6475
I20260620 09:05:40.849526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:40.849858 392 external_mini_cluster.cc:1428] Setting key e191340ebe2e0c10c3bd2763b881b82e
I20260620 09:05:40.852182 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 09:05:40 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:40.858536 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Bootstrap starting.
I20260620 09:05:40.860668 6543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:40.861438 6543 log.cc:824] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:40.863312 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: No bootstrap required, opened a new log
I20260620 09:05:40.863648 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:40.851129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.0.98.62:35565)
0620 09:05:40.851533 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:40.851549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:40.851583 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:05:40.852190 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:40.852206 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:40.852568 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:40.852842 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:40.853711 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:40.854670 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:40.855812 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:40.856248 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:40.856357 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:40.859119 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:40.859124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:40.859135 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:40.859160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:40.860881 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:40.861349 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:40.861355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:40.861367 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:40.861472 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:40.861771 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:40.861774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:40.861776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:40.862152 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:40.862349 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:40.862639 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:05:40.862906 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260620 09:05:40.867640 6543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } }
I20260620 09:05:40.867869 6543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:40.867933 6543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9a82bd647f4e388487436440d43d76, State: Initialized, Role: FOLLOWER
I20260620 09:05:40.868330 6543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } }
I20260620 09:05:40.868479 6543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:40.868544 6543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:40.868646 6543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:40.869612 6543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } }
I20260620 09:05:40.869954 6543 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9a82bd647f4e388487436440d43d76; no voters:
I20260620 09:05:40.870239 6543 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:40.870366 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:40.870602 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9a82bd647f4e388487436440d43d76, State: Running, Role: LEADER
I20260620 09:05:40.870962 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } }
I20260620 09:05:40.871416 6543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:40.872534 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9a82bd647f4e388487436440d43d76. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } }
I20260620 09:05:40.872648 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:40.872726 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } }
I20260620 09:05:40.872813 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
W20260620 09:05:40.875852 6563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:40.875923 6563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:40.876047 6557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:40.876718 6557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:40.881974 6557 catalog_manager.cc:1347] Generated new cluster ID: 8b17336f40834446aad6fd299496890e
I20260620 09:05:40.882050 6557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:40.887746 6557 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:40.888896 6557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:40.895951 6557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Generated new TSK 0
I20260620 09:05:40.896731 6557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:05:40.927366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:41.034775 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:41.035022 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:41.035106 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:41.039095 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:41.039148 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:41.039202 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:41.044341 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:41.045432 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:41.046432 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:41.055948 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:41.055980 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:41.055959 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:41.056311 6569 server_base.cc:1034] running on GCE node
I20260620 09:05:41.056691 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:41.057340 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:41.058519 6569 hybrid_clock.cc:648] HybridClock initialized: now 1781946341058502 us; error 28 us; skew 500 ppm
Jun 20 09:05:41 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946341, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:41.061223 6569 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:05:41.062042 6569 webserver.cc:460] Webserver started at http://127.0.98.1:34627/ using document root <none> and password file <none>
I20260620 09:05:41.062595 6569 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:41.062656 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:41.062820 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:41.064713 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "8baa2262f3b148959fcd48f67a39d9e5"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "1115ea5c24aded5b72b2f5ef0c03daee"
server_key_iv: "512076a70398e0ebf10c719f9f80a14d"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.065268 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8baa2262f3b148959fcd48f67a39d9e5"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "1115ea5c24aded5b72b2f5ef0c03daee"
server_key_iv: "512076a70398e0ebf10c719f9f80a14d"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.068704 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:05:41.070958 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.072229 6569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:05:41.072340 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "8baa2262f3b148959fcd48f67a39d9e5"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "1115ea5c24aded5b72b2f5ef0c03daee"
server_key_iv: "512076a70398e0ebf10c719f9f80a14d"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.072466 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:41.088629 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:41.091840 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:41.091991 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:41.092506 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:41.093370 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:41.093421 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.093466 6569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:41.093523 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.102531 6569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260620 09:05:41.102550 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260620 09:05:41.103458 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 09:05:41 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946341, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:41.113463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6569
I20260620 09:05:41.113601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:41.113857 392 external_mini_cluster.cc:1428] Setting key 3b3fc0760e87c7715898dfc52629f0c4
I20260620 09:05:41.117717 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:41.105248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54347 (local address 127.0.98.62:35565)
0620 09:05:41.105419 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:41.105425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:41.106264 (+ 839us) server_negotiation.cc:407] Connection header received
0620 09:05:41.107580 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:41.107582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:41.107632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:41.107715 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:41.109440 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.109965 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.111032 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.111253 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.111327 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:41.113837 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:41.113841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:41.113843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:41.113864 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:41.115619 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:41.116111 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:41.116117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:41.116122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:41.116219 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:41.116524 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:41.116530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:41.116535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:41.116774 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:41.116882 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:41.117420 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 09:05:41.117549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:05:41.118460 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:41.105694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:54347)
0620 09:05:41.106109 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:41.106151 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:41.107382 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:41.107843 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:41.107849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:41.108187 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:41.109313 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:41.109330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.110051 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.110054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:41.110935 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:41.110943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.111374 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.111376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:41.111526 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:41.112041 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:41.112056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:41.113668 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:41.115722 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:41.115727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:41.115738 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:41.115997 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:41.116306 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:41.116308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:41.116309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:41.116430 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:41.116869 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:41.116878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:41.117187 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 09:05:41.117365 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 09:05:41.117671 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260620 09:05:41.119643 6698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:41.119897 6698 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:41.120476 6698 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:41.122188 6502 ts_manager.cc:194] Registered new tserver with Master: 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:41.123731 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
W20260620 09:05:41.683923 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.2: real 0.568s user 0.001s sys 0.003s
I20260620 09:05:41.684407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:41.792367 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:41.792677 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:41.792755 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:41.796707 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:41.796804 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:41.796897 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:41.802068 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:41.803236 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:41.804293 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:41.813568 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:41.813580 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:41.813620 6703 server_base.cc:1034] running on GCE node
W20260620 09:05:41.813915 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:41.814335 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:41.814970 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:41.816160 6703 hybrid_clock.cc:648] HybridClock initialized: now 1781946341816134 us; error 55 us; skew 500 ppm
Jun 20 09:05:41 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946341, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:41.822845 6703 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:05:41.823750 6703 webserver.cc:460] Webserver started at http://127.0.98.2:36791/ using document root <none> and password file <none>
I20260620 09:05:41.824322 6703 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:41.824410 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:41.824644 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:41.826745 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "218fef6feaef4d58a04db592b8bf87d4"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "3c17ac5e90c47268c4583d7d379b27aa"
server_key_iv: "e309d0ae49265588a91c9d101898581e"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.827346 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "218fef6feaef4d58a04db592b8bf87d4"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "3c17ac5e90c47268c4583d7d379b27aa"
server_key_iv: "e309d0ae49265588a91c9d101898581e"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.831259 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 09:05:41.833554 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.834703 6703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:41.834832 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "218fef6feaef4d58a04db592b8bf87d4"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "3c17ac5e90c47268c4583d7d379b27aa"
server_key_iv: "e309d0ae49265588a91c9d101898581e"
server_key_version: "encryptionkey@0"
I20260620 09:05:41.834935 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:41.844318 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:41.847405 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:41.847599 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:41.848156 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:41.848927 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:41.849014 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.849141 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:41.849202 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:41.859742 6703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38401
I20260620 09:05:41.859788 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38401 every 8 connection(s)
I20260620 09:05:41.860651 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:05:41 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946341, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:41.870441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6703
I20260620 09:05:41.870568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:41.870844 392 external_mini_cluster.cc:1428] Setting key 163d8674baee5842ee7217571db10d80
I20260620 09:05:41.874765 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:41.862483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42411 (local address 127.0.98.62:35565)
0620 09:05:41.862737 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:41.862741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:41.863994 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 09:05:41.864888 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:41.864891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:41.864949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:41.865102 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:41.866268 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.866810 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.867957 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.868192 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.868300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:41.870831 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:41.870834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:41.870836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:41.870857 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:41.872663 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:41.873278 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:41.873283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:41.873285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:41.873367 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:41.873698 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:41.873702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:41.873703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:41.873876 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:41.874003 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:41.874470 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 09:05:41.874589 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260620 09:05:41.875517 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:41.862939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:42411)
0620 09:05:41.863824 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:41.863865 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:41.864684 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:41.865257 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:41.865264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:41.865648 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:41.866125 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:41.866137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.866936 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.866944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:41.867833 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:41.867842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:41.868350 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:41.868353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:41.868517 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:41.869132 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:41.869148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:41.870662 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:41.872795 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:41.872801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:41.872814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:41.873182 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:41.873459 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:41.873462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:41.873464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:41.873601 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:41.874009 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:41.874018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:41.874246 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 09:05:41.874425 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 09:05:41.874722 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":521,"threads_started":1}
I20260620 09:05:41.877009 6832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:41.877292 6832 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:41.877749 6832 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:41.878754 6502 ts_manager.cc:194] Registered new tserver with Master: 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401)
I20260620 09:05:41.879328 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42411
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:05:41.897228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:42.004964 6838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:42.005249 6838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:42.005291 6838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:42.009204 6838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:42.009249 6838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:42.009306 6838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:42.014264 6838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:42.015184 6838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:42.016116 6838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:05:42.025578 6838 server_base.cc:1034] running on GCE node
W20260620 09:05:42.025542 6843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.025533 6844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.025533 6846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.026230 6838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.026973 6838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.028134 6838 hybrid_clock.cc:648] HybridClock initialized: now 1781946342028118 us; error 31 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.031031 6838 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:05:42.031831 6838 webserver.cc:460] Webserver started at http://127.0.98.3:44051/ using document root <none> and password file <none>
I20260620 09:05:42.032385 6838 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.032487 6838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.032693 6838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:42.034710 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "a60b9692ff3f4f038371a1fa72585f61"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "ae9fc909900163e642e0f921bb6c8d29"
server_key_iv: "7359c7febb070a7070dbe54c5df623ef"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.035235 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a60b9692ff3f4f038371a1fa72585f61"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "ae9fc909900163e642e0f921bb6c8d29"
server_key_iv: "7359c7febb070a7070dbe54c5df623ef"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.039451 6838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:05:42.041852 6853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.043040 6838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:42.043177 6838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "a60b9692ff3f4f038371a1fa72585f61"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "ae9fc909900163e642e0f921bb6c8d29"
server_key_iv: "7359c7febb070a7070dbe54c5df623ef"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.043286 6838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.058578 6838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.061894 6838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:42.062090 6838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.062683 6838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:42.063548 6838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:42.063618 6838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.063689 6838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:42.063741 6838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.072415 6838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43077
I20260620 09:05:42.072435 6966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43077 every 8 connection(s)
I20260620 09:05:42.073434 6838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:42.083087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6838
I20260620 09:05:42.083196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:42.083490 392 external_mini_cluster.cc:1428] Setting key 84b5e323ba2b49cc68cad30b9146a703
I20260620 09:05:42.086884 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.075240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59269 (local address 127.0.98.62:35565)
0620 09:05:42.075373 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.075378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.076253 (+ 875us) server_negotiation.cc:407] Connection header received
0620 09:05:42.077773 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.077776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.077828 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.077912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.079001 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.079526 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.080642 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.080885 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.080989 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.083381 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.083385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.083387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.083414 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.085135 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.085615 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.085618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.085619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.085697 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.085970 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.085972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.085973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.086138 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.086215 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.086656 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.086761 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:05:42.087706 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.075618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:59269)
0620 09:05:42.076077 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.076140 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.077546 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.078068 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.078074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.078410 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.078877 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.078889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.079641 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.079643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.080533 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.080541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.081029 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.081032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.081235 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.081758 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.081772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.083222 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.085233 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.085237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.085248 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.085515 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.085759 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.085761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.085762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.085889 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.086222 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.086227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.086444 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.086604 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.086842 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260620 09:05:42.088821 6967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:42.089074 6967 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:42.089541 6967 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:42.090514 6502 ts_manager.cc:194] Registered new tserver with Master: a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077)
I20260620 09:05:42.091086 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59269
I20260620 09:05:42.097355 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 09:05:42.123287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46867
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35565,127.0.98.61:46867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:05:42.126016 6698 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
W20260620 09:05:42.235224 6972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:42.235468 6972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:42.235507 6972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:42.239511 6972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:42.239562 6972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:42.239583 6972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:42.239599 6972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:42.239614 6972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:42.244637 6972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35565,127.0.98.61:46867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
W20260620 09:05:42.244802 6972 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35565,127.0.98.61:46867'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 09:05:42.245692 6972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:42.246711 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:42.254765 6977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.254868 6978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.254882 6972 server_base.cc:1034] running on GCE node
W20260620 09:05:42.254761 6980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.255595 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.256654 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.257802 6972 hybrid_clock.cc:648] HybridClock initialized: now 1781946342257788 us; error 29 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.260581 6972 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:05:42.261426 6972 webserver.cc:460] Webserver started at http://127.0.98.61:40355/ using document root <none> and password file <none>
I20260620 09:05:42.261945 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.262002 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.262159 6972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:42.264070 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/instance:
uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "21a3dbaa3df8c81fd8ac8e3b16dfda82"
server_key_iv: "eba1fd28df948cc39a11c3a154fca6db"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.264575 6972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal/instance:
uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "21a3dbaa3df8c81fd8ac8e3b16dfda82"
server_key_iv: "eba1fd28df948cc39a11c3a154fca6db"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.268051 6972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:42.270363 6987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.271687 6972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:42.271809 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "21a3dbaa3df8c81fd8ac8e3b16dfda82"
server_key_iv: "eba1fd28df948cc39a11c3a154fca6db"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.271910 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.294277 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.297605 6972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:42.297829 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.304409 6972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46867
I20260620 09:05:42.304687 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46867 every 8 connection(s)
I20260620 09:05:42.305409 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:05:42.308476 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:42.309233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 6972
I20260620 09:05:42.313284 7040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:42.323380 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.310690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.98.61:46867)
0620 09:05:42.311067 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.311084 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.311125 (+ 41us) server_negotiation.cc:407] Connection header received
0620 09:05:42.311811 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.311834 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.312238 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.312582 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.313246 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.314642 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.315772 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.316230 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.316395 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.318868 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.318878 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.318886 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.318910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.320729 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.321201 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.321213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.321218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.321352 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.321642 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.321645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.321647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.321993 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.322180 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.322449 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.322695 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260620 09:05:42.324389 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.315160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.98.62:35565)
0620 09:05:42.315303 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.315308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.315587 (+ 279us) server_negotiation.cc:407] Connection header received
0620 09:05:42.315708 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.315711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.315761 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.315849 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.316497 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.317007 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.317948 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.318169 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.318278 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.320909 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.320912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.320914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.320936 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.322461 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.323052 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.323057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.323059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.323141 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.323435 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.323439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.323441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.323688 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.323796 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.324092 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.324226 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:42.324414 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.315107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36126)
0620 09:05:42.315466 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.315501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.315627 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.315894 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.315900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.316245 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.316369 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.316374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.317172 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.317175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.317805 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.317813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.318309 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.318313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.318440 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.318992 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.319005 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.320757 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.322720 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.322729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.322735 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.322953 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.323231 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.323236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.323238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.323324 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.323792 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.323797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.323907 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.324041 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.324210 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260620 09:05:42.325536 7040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:05:42.332621 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.325923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37528)
0620 09:05:42.326040 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.326055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.326125 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.326344 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.326347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.326492 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.326564 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.326569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.327585 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.327588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.328089 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.328096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.328521 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.328525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.328609 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.328980 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.328990 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.330154 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.331567 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.331569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.331570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.331734 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.331940 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.331942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.331943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.332006 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.332279 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.332280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.332349 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.332422 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.332503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:05:42.332708 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.326003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.98.61:46867)
0620 09:05:42.326109 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.326113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.326124 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:42.326206 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.326210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.326241 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.326323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.326667 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.327475 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.328195 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.328421 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.328528 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.330280 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.330283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.330285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.330304 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.331483 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.331808 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.331810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.331811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.331871 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.332074 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.332076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.332077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.332208 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.332274 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.332473 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.332549 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:05:42.335179 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Bootstrap starting.
I20260620 09:05:42.337404 7040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:42.338130 7040 log.cc:824] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:42.339892 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: No bootstrap required, opened a new log
I20260620 09:05:42.342772 7040 raft_consensus.cc:348] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } }
I20260620 09:05:42.342985 7040 raft_consensus.cc:374] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:42.343058 7040 raft_consensus.cc:729] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e854d9bf26b54fbea9b307e4dd4bce6b, State: Initialized, Role: FOLLOWER
I20260620 09:05:42.343461 7040 consensus_queue.cc:260] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } }
I20260620 09:05:42.343868 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } } }
I20260620 09:05:42.343969 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:42.344530 7040 sys_catalog.cc:564] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:05:42.349179 7057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:42.349255 7057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:42.351066 6972 master_runner.cc:428] Detected that this master e854d9bf26b54fbea9b307e4dd4bce6b is joining an existing cluster
I20260620 09:05:42.351122 6972 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46867
I20260620 09:05:42.361429 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.353715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.0.98.61:46867)
0620 09:05:42.353802 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.353808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.353844 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:05:42.354000 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.354004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.354044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.354122 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.354573 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.355362 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.356593 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.356918 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.357025 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.358136 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.358140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.358142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.358162 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.359776 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360209 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.360215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.360217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.360307 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360576 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.360579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.360581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.360774 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.360873 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.361134 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.361241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":37}
I20260620 09:05:42.361744 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.353319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.98.62:35565)
0620 09:05:42.353648 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.353654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.353710 (+ 56us) server_negotiation.cc:407] Connection header received
0620 09:05:42.353889 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.353892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.353956 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.354020 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.354648 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.355443 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.356333 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.356593 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.356690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.358104 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.358111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.358116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.358142 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.359795 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360329 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.360335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.360340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.360422 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360705 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.360709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.360711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.360925 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.361076 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.361309 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.361460 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260620 09:05:42.361893 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.353492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37532)
0620 09:05:42.353777 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.353789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.353898 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.354132 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.354135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.354377 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.354468 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.354473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.355485 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.355489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.356480 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.356490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.357023 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.357026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.357176 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.357654 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.357669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.358013 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.359885 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.359891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.359895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.360108 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.360388 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.360393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.360473 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.360864 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.360867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.360951 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.361620 (+ 669us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.361724 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":25,"thread_start_us":78,"threads_started":1}
I20260620 09:05:42.362125 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.353290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36132)
0620 09:05:42.353632 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.353647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.353778 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.354170 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.354173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.354429 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.354548 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.354556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.355521 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.355527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.356216 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.356223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.356719 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.356723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.356887 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.357461 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.357477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.357943 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.359917 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.359920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.359922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.360233 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.360524 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.360526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.360528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.360610 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.361035 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.361060 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.361136 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.361230 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.361336 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":122,"thread_start_us":117,"threads_started":1}
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:05:42.375360 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.366101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.0.98.61:46867)
0620 09:05:42.366212 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.366216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.366802 (+ 586us) server_negotiation.cc:407] Connection header received
0620 09:05:42.366956 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.366959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.367016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.367101 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.367723 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.368468 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.369382 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.369707 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.369806 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.372215 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.372219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.372222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.372244 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.373708 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.374201 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.374204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.374205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.374267 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.374521 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.374523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.374524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.374664 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.374732 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.375107 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.375228 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:05:42.375376 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.366033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37534)
0620 09:05:42.366698 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.366733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.366837 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.367170 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.367175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.367518 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.367614 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.367620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.368572 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.368574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.369280 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.369289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.369809 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.369812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.369881 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.370365 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.370376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.372091 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.373803 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.373811 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.373817 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.374106 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.374346 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.374349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.374351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.374438 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.374757 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.374763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.374879 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.375029 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.375200 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":94,"threads_started":1}
I20260620 09:05:42.376932 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46867
}
attrs {
promote: true
}
}
dest_uuid: "7e9a82bd647f4e388487436440d43d76"
cas_config_opid_index: -1
I20260620 09:05:42.377604 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } }
I20260620 09:05:42.379335 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } }
I20260620 09:05:42.380074 7014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:42.380916 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46867
I20260620 09:05:42.381229 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } }
I20260620 09:05:42.381326 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:42.381371 6972 master.cc:537] Master@127.0.98.61:46867 shutting down...
I20260620 09:05:42.381781 7014 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 09:05:42.382373 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:05:42.384008 6491 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 09:05:42.385274 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 -> Peer e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61:46867): Couldn't send request to peer e854d9bf26b54fbea9b307e4dd4bce6b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 09:05:42.387161 6972 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:42.387465 6972 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:42.387565 6972 tablet_replica.cc:330] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260620 09:05:42.394321 6972 master.cc:552] Master@127.0.98.61:46867 shutdown complete.
I20260620 09:05:42.394376 6972 master_runner.cc:305] Clearing existing system tablet
I20260620 09:05:42.394637 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.394702 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.396238 6972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 09:05:42.397185 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.397418 6972 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.397528 6972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:42.397558 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "21a3dbaa3df8c81fd8ac8e3b16dfda82"
server_key_iv: "eba1fd28df948cc39a11c3a154fca6db"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.399577 6972 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:42.402478 6972 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:05:42.402565 6972 log.cc:1196] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 09:05:42.402801 6972 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Deleting consensus metadata
I20260620 09:05:42.403541 6972 master_runner.cc:315] Copying system tablet from 127.0.98.62:35565
I20260620 09:05:42.404696 6972 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 09:05:42.407902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42597
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:05:42.414155 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.405403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.0.98.62:35565)
0620 09:05:42.405549 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.405554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.405764 (+ 210us) server_negotiation.cc:407] Connection header received
0620 09:05:42.405919 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.405922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.405982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.406083 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.406565 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.407277 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.408405 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.408640 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.408749 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.410042 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.410049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.410053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.410087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.412117 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.412623 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.412629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.412633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.412727 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.413025 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.413031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.413036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.413356 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.413466 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.413759 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.413907 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:05:42.414316 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36142)
0620 09:05:42.405674 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.405686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.405811 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.406100 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.406103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.406302 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.406434 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.406444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.407430 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.407433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.408087 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.408100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.408763 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.408766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.408858 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.409478 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.409493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.409885 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.412241 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.412245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.412247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.412519 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.412828 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.412831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.412833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.412923 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.413472 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.413475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.413573 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.413674 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.413820 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":99,"threads_started":1}
I20260620 09:05:42.415063 6527 tablet_copy_service.cc:140] P 7e9a82bd647f4e388487436440d43d76: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e854d9bf26b54fbea9b307e4dd4bce6b ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36142)
I20260620 09:05:42.415273 6527 tablet_copy_service.cc:161] P 7e9a82bd647f4e388487436440d43d76: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e854d9bf26b54fbea9b307e4dd4bce6b at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36142: session id = e854d9bf26b54fbea9b307e4dd4bce6b-00000000000000000000000000000000
I20260620 09:05:42.417555 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:05:42.419322 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:05:42.421600 6972 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: tablet copy: Starting download of 0 data blocks...
I20260620 09:05:42.421792 6972 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: tablet copy: Starting download of 1 WAL segments...
I20260620 09:05:42.423408 6972 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:05:42.425143 6527 tablet_copy_service.cc:342] P 7e9a82bd647f4e388487436440d43d76: Request end of tablet copy session e854d9bf26b54fbea9b307e4dd4bce6b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36142
I20260620 09:05:42.425247 6527 tablet_copy_service.cc:433] P 7e9a82bd647f4e388487436440d43d76: ending tablet copy session e854d9bf26b54fbea9b307e4dd4bce6b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e854d9bf26b54fbea9b307e4dd4bce6b
I20260620 09:05:42.432830 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:42.434448 7082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.434584 6972 server_base.cc:1034] running on GCE node
W20260620 09:05:42.437846 7085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.437911 7083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.438238 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.438532 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.439667 6972 hybrid_clock.cc:648] HybridClock initialized: now 1781946342439654 us; error 30 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.441287 6972 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 09:05:42.441716 6972 webserver.cc:460] Webserver started at http://127.0.98.61:45461/ using document root <none> and password file <none>
I20260620 09:05:42.441892 6972 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.441933 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.443269 6972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:42.444221 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.444443 6972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:42.444511 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "21a3dbaa3df8c81fd8ac8e3b16dfda82"
server_key_iv: "eba1fd28df948cc39a11c3a154fca6db"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.444582 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.461835 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.464231 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.468757 6972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46867
I20260620 09:05:42.468791 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46867 every 8 connection(s)
I20260620 09:05:42.469339 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:05:42.471194 7145 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:05:42.472254 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Bootstrap starting.
I20260620 09:05:42.482599 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:05:42.482945 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Bootstrap complete.
I20260620 09:05:42.483619 7145 raft_consensus.cc:348] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } }
I20260620 09:05:42.483753 7145 raft_consensus.cc:729] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e854d9bf26b54fbea9b307e4dd4bce6b, State: Initialized, Role: LEARNER
I20260620 09:05:42.483851 7145 consensus_queue.cc:260] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } }
I20260620 09:05:42.484186 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } }
I20260620 09:05:42.484282 7145 sys_catalog.cc:564] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:42.484387 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: LEARNER
I20260620 09:05:42.487805 7159 catalog_manager.cc:1259] Loaded cluster ID: 8b17336f40834446aad6fd299496890e
I20260620 09:05:42.487864 7159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: loading cluster ID for follower catalog manager: success
I20260620 09:05:42.489959 7159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: acquiring CA information for follower catalog manager: success
I20260620 09:05:42.491259 7159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 09:05:42.538491 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:42.538753 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:42.538821 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:42.542889 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:42.542938 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:42.542958 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:42.543006 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:42.543038 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:42.548470 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:42.549484 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:42.550441 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:42.558507 7164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.558516 7167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.558509 7165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.558732 7081 server_base.cc:1034] running on GCE node
I20260620 09:05:42.559346 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.560415 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.561589 7081 hybrid_clock.cc:648] HybridClock initialized: now 1781946342561557 us; error 45 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.564118 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:05:42.564934 7081 webserver.cc:460] Webserver started at http://127.0.98.60:45601/ using document root <none> and password file <none>
I20260620 09:05:42.565497 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.565583 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.565802 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:42.567773 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/instance:
uuid: "fa7f039bb827416a8f96302cc02164a0"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "e6cfc4ceb54281478e67b3f67e04f6dc"
server_key_iv: "383f6b438a2de1f0fb02ecceb37a332d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.568329 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal/instance:
uuid: "fa7f039bb827416a8f96302cc02164a0"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "e6cfc4ceb54281478e67b3f67e04f6dc"
server_key_iv: "383f6b438a2de1f0fb02ecceb37a332d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.571975 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:05:42.574476 7174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.575650 7081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:42.575778 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "fa7f039bb827416a8f96302cc02164a0"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "e6cfc4ceb54281478e67b3f67e04f6dc"
server_key_iv: "383f6b438a2de1f0fb02ecceb37a332d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.575877 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.593786 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.596899 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:42.597138 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.604080 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42597
I20260620 09:05:42.604131 7226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42597 every 8 connection(s)
I20260620 09:05:42.605007 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:05:42.607681 7227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:42.611614 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:05:42.614965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7081
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:05:42.624639 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.613565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.0.98.62:35565)
0620 09:05:42.613689 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.613693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.614385 (+ 692us) server_negotiation.cc:407] Connection header received
0620 09:05:42.615296 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.615299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.615362 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.615453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.616698 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.617286 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.618313 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.618528 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.618633 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.621154 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.621159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.621166 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.621188 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.622714 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.623172 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.623177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.623179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.623256 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.623572 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.623575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.623577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.623741 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.623839 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.624338 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.624462 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:42.625602 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.613869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36148)
0620 09:05:42.614253 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.614291 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.615096 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.615615 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.615622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.616109 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.616562 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.616574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.617406 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.617410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.618197 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.618205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.618663 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.618667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.618824 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.619391 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.619405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.620992 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.622814 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.622823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.622832 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.623075 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.623339 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.623344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.623346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.623475 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.623836 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.623842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.624097 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.624300 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.624588 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"spinlock_wait_cycles":3840,"thread_start_us":91,"threads_started":1}
I20260620 09:05:42.625779 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.615723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.98.60:42597)
0620 09:05:42.616087 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.616109 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.616128 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:05:42.616161 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.616164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.616318 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.616480 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.617234 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.618067 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.618939 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.619430 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.619553 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.622108 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.622113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.622122 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.622142 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.623642 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.624099 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.624104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.624105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.624176 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.624467 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.624472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.624474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.624701 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.624950 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.625487 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.625620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":150,"threads_started":1}
I20260620 09:05:42.626857 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:05:42.634516 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.627335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37548)
0620 09:05:42.627666 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.627679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.627739 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.627945 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.627946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.628067 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.628143 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.628147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.628978 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.628980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.629580 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.629587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.630071 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.630073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.630140 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.630548 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.630557 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.631660 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.633404 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.633406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.633408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.633572 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.633795 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.633797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.633798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.633862 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.634171 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.634173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.634236 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.634327 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.634410 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260620 09:05:42.634637 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.627452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.0.98.61:46867)
0620 09:05:42.627729 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.627733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.627745 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:42.627802 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.627804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.627848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.627920 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.628235 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.628886 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.629685 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.629985 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.630064 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.631777 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.631779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.631781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.631804 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.633317 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.633649 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.633651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.633652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.633723 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.633944 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.633946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.633947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.634097 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.634169 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.634391 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.634486 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260620 09:05:42.635296 7227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:05:42.642944 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.635738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49318)
0620 09:05:42.635835 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.635846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.635927 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.636163 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.636165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.636300 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.636351 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.636355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.637071 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.637074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.637571 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.637577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.638535 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.638538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.638609 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.638991 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.639001 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.640167 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.641606 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.641612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.641615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.641784 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.642039 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.642042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.642044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.642113 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.642477 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.642492 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.642565 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.642663 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.642783 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:05:42.642951 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.635843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.98.60:42597)
0620 09:05:42.635933 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.635937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.635950 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:42.636032 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.636035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.636076 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.636154 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.636451 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.636932 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.637704 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.638427 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.638539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.640295 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.640299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.640300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.640318 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.641478 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.641873 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.641879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.641881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.641947 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.642216 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.642219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.642221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.642381 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.642498 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.642714 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.642835 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":37}
I20260620 09:05:42.645882 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Bootstrap starting.
I20260620 09:05:42.648260 7227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:42.648984 7227 log.cc:824] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:42.650799 7227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: No bootstrap required, opened a new log
I20260620 09:05:42.653839 7227 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } }
I20260620 09:05:42.654049 7227 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:42.654121 7227 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7f039bb827416a8f96302cc02164a0, State: Initialized, Role: FOLLOWER
I20260620 09:05:42.654590 7227 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } }
I20260620 09:05:42.655033 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } } }
I20260620 09:05:42.655180 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:42.655802 7227 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:05:42.660621 7244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:42.660712 7244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:42.663641 7081 master_runner.cc:428] Detected that this master fa7f039bb827416a8f96302cc02164a0 is joining an existing cluster
I20260620 09:05:42.663702 7081 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42597
I20260620 09:05:42.678719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6569
I20260620 09:05:42.685024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33591
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35565
--webserver_port=34627
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:05:42.694023 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37558)
0620 09:05:42.666697 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.666711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.666840 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.667197 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.667199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.667541 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.667817 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.667824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668846 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.668848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.670625 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.670636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.671267 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.671270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.671333 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.671812 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.671824 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.689661 (+ 17837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.692377 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.692380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.692383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.692621 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.692895 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.692897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.692900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.693005 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.693492 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.693496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.693589 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.693673 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.693821 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":217,"spinlock_wait_cycles":19584,"thread_start_us":79,"threads_started":1}
I20260620 09:05:42.694651 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.666132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.98.61:46867)
0620 09:05:42.666345 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.666349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.666779 (+ 430us) server_negotiation.cc:407] Connection header received
0620 09:05:42.666957 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.666960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.667010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.667057 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.667960 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668713 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.670746 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.671112 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.671223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.689870 (+ 18647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.689874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.689877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.689911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.692252 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.692725 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.692729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.692730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.692806 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.693133 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.693137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.693138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.693358 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.693441 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.694369 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.694501 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 09:05:42.696698 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.666004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.0.98.62:35565)
0620 09:05:42.666259 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.666264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.666574 (+ 310us) server_negotiation.cc:407] Connection header received
0620 09:05:42.666745 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.666748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.666798 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.666849 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.667525 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668337 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.669579 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.669907 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.669984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.690694 (+ 20710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.690699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.690702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.690751 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.694278 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.695348 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.695352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.695353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.695436 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.695764 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.695767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.695769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.695983 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.696063 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.696409 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.696541 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260620 09:05:42.697155 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.666436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.98.60:42597)
0620 09:05:42.666544 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.666549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.666775 (+ 226us) server_negotiation.cc:407] Connection header received
0620 09:05:42.666925 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.666931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.666967 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.667011 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.667937 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668677 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.669973 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.670935 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.670990 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.691168 (+ 20178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.691172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.691175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.691194 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.693427 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.694803 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.694807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.694809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.694872 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.696134 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.696140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.696144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.696378 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.696472 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.696831 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.696937 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:05:42.697459 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36162)
0620 09:05:42.666495 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.666509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.666638 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.666987 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.666992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.667210 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.667379 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.667387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668486 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.668490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.669438 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.669448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.670147 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.670152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.670239 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.671384 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.671399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.690524 (+ 19125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.695004 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.695007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.695009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.695232 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.695557 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.695561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.695563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.695648 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.696136 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.696139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.696237 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.696881 (+ 644us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.697026 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"spinlock_wait_cycles":10368,"thread_start_us":129,"threads_started":1}
I20260620 09:05:42.697476 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.666425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49330)
0620 09:05:42.666697 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.666711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.666829 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.667206 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.667208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.667536 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.667819 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.667824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.668800 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.668803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.669856 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.669865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.671087 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.671089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.671144 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.689643 (+ 18499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.689667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.691050 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.694436 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.694440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.694442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.694693 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.695904 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.695909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.695914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.696024 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.696541 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.696546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.696647 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.697094 (+ 447us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.697245 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":155,"spinlock_wait_cycles":7424,"thread_start_us":83,"threads_started":1}
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946340, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:05:42.710383 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.701463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49332)
0620 09:05:42.701583 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.701599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.701683 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.701977 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.701980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.702162 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.702264 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.702270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.703239 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.703243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.704162 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.704177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.704735 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.704738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.704838 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.705319 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.705333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.706881 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.708819 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.708822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.708824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.709106 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.709376 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.709379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.709381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.709478 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.709874 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.709877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.709970 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.710076 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.710208 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 09:05:42.710386 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.701538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.0.98.60:42597)
0620 09:05:42.701644 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.701649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.701661 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:42.701817 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.701820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.701869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.701942 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.702376 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.703132 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.704291 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.704623 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.704723 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.707010 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.707013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.707016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.707040 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.708619 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.709199 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.709202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.709204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.709282 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.709569 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.709572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.709574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.709776 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.709869 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.710139 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.710241 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:05:42.711637 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42597: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fa7f039bb827416a8f96302cc02164a0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42597
}
attrs {
promote: true
}
}
dest_uuid: "7e9a82bd647f4e388487436440d43d76"
cas_config_opid_index: 5
I20260620 09:05:42.712245 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } }
I20260620 09:05:42.713286 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fa7f039bb827416a8f96302cc02164a0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.714171 7201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:42.714640 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42597
I20260620 09:05:42.714792 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.714891 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:42.715485 7201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:05:42.716089 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:42.716435 7081 master.cc:537] Master@127.0.98.60:42597 shutting down...
W20260620 09:05:42.717667 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 -> Peer fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597): Couldn't send request to peer fa7f039bb827416a8f96302cc02164a0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 09:05:42.721099 7081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:42.721365 7081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:42.721458 7081 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: stopping tablet replica
I20260620 09:05:42.722724 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.713423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37564)
0620 09:05:42.717954 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.717968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.718075 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.718453 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.718455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.718482 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:42.718561 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.718566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.719686 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.719693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.720916 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.720929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.722327 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.722331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.722429 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.722470 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.722543 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.722607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4473}
I20260620 09:05:42.723119 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.713653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.0.98.61:46867)
0620 09:05:42.717875 (+ 4222us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.717880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.718152 (+ 272us) server_negotiation.cc:407] Connection header received
0620 09:05:42.718203 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.718206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.718261 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.718340 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:42.718685 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.719461 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.721094 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.722044 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.722140 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.722268 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.722938 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.723011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4104}
I20260620 09:05:42.723951 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 LEARNER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 09:05:42.724449 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 09:05:42.726320 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fa7f039bb827416a8f96302cc02164a0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.727840 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.728015 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: LEARNER
I20260620 09:05:42.728355 7068 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: attempting to promote NON_VOTER e854d9bf26b54fbea9b307e4dd4bce6b to VOTER
I20260620 09:05:42.728571 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9a82bd647f4e388487436440d43d76. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: true } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.728684 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: LEARNER
I20260620 09:05:42.728786 7081 master.cc:552] Master@127.0.98.60:42597 shutdown complete.
I20260620 09:05:42.728775 7068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } }
I20260620 09:05:42.728847 7081 master_runner.cc:305] Clearing existing system tablet
I20260620 09:05:42.729143 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.729219 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.730086 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.729809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49344)
0620 09:05:42.729938 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.730010 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:05:42.730352 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 LEARNER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 09:05:42.730605 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:42.730978 7256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 09:05:42.731092 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.730681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49356)
0620 09:05:42.730914 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.730965 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260620 09:05:42.731568 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 -> Peer fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597): Couldn't send request to peer fa7f039bb827416a8f96302cc02164a0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:05:42.732654 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.733090 7118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.734012 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.734102 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:42.734213 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.734274 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:42.734447 7148 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.734568 7148 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:42.738555 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.738776 7081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.738965 7081 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260620 09:05:42.739009 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "fa7f039bb827416a8f96302cc02164a0"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "e6cfc4ceb54281478e67b3f67e04f6dc"
server_key_iv: "383f6b438a2de1f0fb02ecceb37a332d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.741187 7081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:42.743453 7081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 09:05:42.743541 7081 log.cc:1196] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 09:05:42.743770 7081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Deleting consensus metadata
I20260620 09:05:42.743918 7081 master_runner.cc:315] Copying system tablet from 127.0.98.62:35565
I20260620 09:05:42.745086 7081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35565
I20260620 09:05:42.757987 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.745659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.98.62:35565)
0620 09:05:42.745767 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.745771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.749349 (+ 3578us) server_negotiation.cc:407] Connection header received
0620 09:05:42.749465 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.749469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.749531 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.749624 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.750153 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.751087 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.752209 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.752523 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.752627 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.754321 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.754325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.754328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.754354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.756161 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.756693 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.756697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.756699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.756786 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.757126 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.757129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.757131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.757333 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.757506 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.757715 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.757823 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:05:42.757997 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.745573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36176)
0620 09:05:42.749248 (+ 3675us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.749264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.749379 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.749666 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.749668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.749883 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.750010 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.750017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.751207 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.751211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.752091 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.752101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.752641 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.752645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.752738 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.753761 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.753775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.754166 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.756279 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.756282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.756285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.756591 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.756910 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.756913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.756915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.757006 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.757449 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.757452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.757556 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.757651 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.757789 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3575,"thread_start_us":84,"threads_started":1}
I20260620 09:05:42.758818 6527 tablet_copy_service.cc:140] P 7e9a82bd647f4e388487436440d43d76: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fa7f039bb827416a8f96302cc02164a0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36176)
I20260620 09:05:42.758920 6527 tablet_copy_service.cc:161] P 7e9a82bd647f4e388487436440d43d76: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fa7f039bb827416a8f96302cc02164a0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36176: session id = fa7f039bb827416a8f96302cc02164a0-00000000000000000000000000000000
I20260620 09:05:42.759997 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 09:05:42.761253 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 09:05:42.763787 7081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: tablet copy: Starting download of 0 data blocks...
I20260620 09:05:42.764031 7081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: tablet copy: Starting download of 1 WAL segments...
I20260620 09:05:42.765537 7081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 09:05:42.767195 6527 tablet_copy_service.cc:342] P 7e9a82bd647f4e388487436440d43d76: Request end of tablet copy session fa7f039bb827416a8f96302cc02164a0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36176
I20260620 09:05:42.767277 6527 tablet_copy_service.cc:433] P 7e9a82bd647f4e388487436440d43d76: ending tablet copy session fa7f039bb827416a8f96302cc02164a0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fa7f039bb827416a8f96302cc02164a0
I20260620 09:05:42.774931 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:42.777073 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.782037 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.782344 7081 server_base.cc:1034] running on GCE node
W20260620 09:05:42.782445 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.782662 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.783002 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.784140 7081 hybrid_clock.cc:648] HybridClock initialized: now 1781946342784119 us; error 39 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.785966 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 09:05:42.786497 7081 webserver.cc:460] Webserver started at http://127.0.98.60:40683/ using document root <none> and password file <none>
I20260620 09:05:42.786746 7081 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.786796 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.788244 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 09:05:42.789287 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.789593 7081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:42.789670 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "fa7f039bb827416a8f96302cc02164a0"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "e6cfc4ceb54281478e67b3f67e04f6dc"
server_key_iv: "383f6b438a2de1f0fb02ecceb37a332d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.789773 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.812844 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.815104 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.820029 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42597
I20260620 09:05:42.820056 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42597 every 8 connection(s)
I20260620 09:05:42.820706 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:05:42.822538 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:05:42.823678 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Bootstrap starting.
W20260620 09:05:42.834054 7253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:42.834317 7253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:42.834390 7253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 09:05:42.834779 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:05:42.835062 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: Bootstrap complete.
I20260620 09:05:42.835618 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } }
I20260620 09:05:42.835749 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa7f039bb827416a8f96302cc02164a0, State: Initialized, Role: LEARNER
I20260620 09:05:42.835847 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } }
I20260620 09:05:42.836161 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:42.836269 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:42.836350 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: LEARNER
W20260620 09:05:42.839864 7253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260620 09:05:42.839895 7346 catalog_manager.cc:1259] Loaded cluster ID: 8b17336f40834446aad6fd299496890e
I20260620 09:05:42.839941 7346 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: loading cluster ID for follower catalog manager: success
W20260620 09:05:42.839955 7253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:42.840040 7253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:42.841980 7346 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: acquiring CA information for follower catalog manager: success
I20260620 09:05:42.843292 7346 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:05:42.846508 7253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34627
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:42.847580 7253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:42.848806 7253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:42.860808 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.860810 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:42.861027 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:42.861639 7253 server_base.cc:1034] running on GCE node
I20260620 09:05:42.862036 7253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:42.862649 7253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:42.863837 7253 hybrid_clock.cc:648] HybridClock initialized: now 1781946342863814 us; error 38 us; skew 500 ppm
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:42.866490 7253 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:05:42.867337 7253 webserver.cc:460] Webserver started at http://127.0.98.1:34627/ using document root <none> and password file <none>
I20260620 09:05:42.867841 7253 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:42.867926 7253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:42.871780 7253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:05:42.873879 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.875082 7253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:42.875198 7253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "8baa2262f3b148959fcd48f67a39d9e5"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "1115ea5c24aded5b72b2f5ef0c03daee"
server_key_iv: "512076a70398e0ebf10c719f9f80a14d"
server_key_version: "encryptionkey@0"
I20260620 09:05:42.875715 7253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:42.881326 6832 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
I20260620 09:05:42.896751 7253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:42.899571 7253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:42.899765 7253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:42.900373 7253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:42.901207 7253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:42.901250 7253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.901319 7253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:42.901350 7253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:42.910594 7253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260620 09:05:42.910626 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260620 09:05:42.911542 7253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:42.911769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7253
I20260620 09:05:42.911969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6703
I20260620 09:05:42.919334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38401
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35565
--webserver_port=36791
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 09:05:42 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946342, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:05:42.929872 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.915090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55291 (local address 127.0.98.62:35565)
0620 09:05:42.915214 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.915218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.915237 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:05:42.915986 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.915993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.916052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.916128 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.918035 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.918875 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921028 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921367 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921442 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.925502 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.925505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.925511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.925544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.928020 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928493 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.928572 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928871 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.929096 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.929170 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.929596 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.929708 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:42.929872 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.915387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36349 (local address 127.0.98.61:46867)
0620 09:05:42.915527 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.915533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.916240 (+ 707us) server_negotiation.cc:407] Connection header received
0620 09:05:42.916413 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.916416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.916468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.916520 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.918248 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.919045 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921028 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921368 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921442 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.925326 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.925330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.925333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.925359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.927410 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928020 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.928116 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928491 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.928736 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.928809 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.929573 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.929708 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:42.929982 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.913401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46727 (local address 127.0.98.60:42597)
0620 09:05:42.913681 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:42.913688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:42.915787 (+ 2099us) server_negotiation.cc:407] Connection header received
0620 09:05:42.915968 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:42.915974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:42.916046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:42.916132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:42.918271 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.919454 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921095 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921489 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.925242 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:42.925248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:42.925253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:42.925290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:42.927225 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928010 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.928119 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928521 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:42.928526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:42.928530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:42.928785 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:42.928866 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:42.929578 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 09:05:42.929738 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260620 09:05:42.931690 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.913898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:55291)
0620 09:05:42.914494 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.914549 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.915737 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.916535 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.916544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.917258 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.917888 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.917900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.919078 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.919081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.920893 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.920900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921654 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.921937 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.922733 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.922752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.925346 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.928141 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.928150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.928370 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.928682 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.928687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.928767 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.929243 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.929246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.929324 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.930479 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.930776 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":835,"spinlock_wait_cycles":8064,"thread_start_us":647,"threads_started":1}
I20260620 09:05:42.931970 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.913898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.98.1:46727)
0620 09:05:42.915698 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.915714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.915856 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.916533 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.916541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.917697 (+ 1156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.917899 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.917910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.919590 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.919596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.920918 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.920932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921791 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.921958 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.922535 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.922561 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.925003 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.927387 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.927394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.927412 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.927871 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.928246 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.928255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.928399 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.928960 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.928966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.929270 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.930186 (+ 916us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.930700 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"mutex_wait_us":283,"spinlock_wait_cycles":11264,"thread_start_us":177,"threads_started":1}
I20260620 09:05:42.931708 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:42.915082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.98.1:36349)
0620 09:05:42.916154 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:42.916167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:42.916303 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:42.916645 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:42.916647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:42.917259 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:42.918121 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.918131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.919182 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.919184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.920889 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:42.920900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:42.921654 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:42.921658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:42.921941 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:42.923015 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:42.923030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:42.925161 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:42.927540 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.927545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.927548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.927889 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.928224 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:42.928227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:42.928230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:42.928385 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:42.928887 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:42.928893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:42.929272 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 09:05:42.930186 (+ 914us) client_negotiation.cc:240] Negotiation successful
0620 09:05:42.930590 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":778,"spinlock_wait_cycles":42752,"thread_start_us":69,"threads_started":1}
I20260620 09:05:42.933158 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42597
I20260620 09:05:42.933220 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46867
I20260620 09:05:42.933244 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:42.933423 7473 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:42.933422 7474 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:42.933422 7472 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:42.934121 7472 heartbeater.cc:511] Master 127.0.98.60:42597 requested a full tablet report, sending...
I20260620 09:05:42.934121 7473 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:42.934185 7474 heartbeater.cc:511] Master 127.0.98.61:46867 requested a full tablet report, sending...
I20260620 09:05:42.935345 6502 ts_manager.cc:194] Re-registered known tserver with Master: 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:42.935741 7109 ts_manager.cc:194] Registered new tserver with Master: 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:42.935741 7293 ts_manager.cc:194] Registered new tserver with Master: 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:42.936079 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55291
W20260620 09:05:43.033918 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:43.034152 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:43.034192 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:43.038220 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:43.038275 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:43.038331 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:43.043355 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36791
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:43.044335 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:43.045436 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:43.054948 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:43.055083 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:43.054960 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:43.055158 7480 server_base.cc:1034] running on GCE node
I20260620 09:05:43.055632 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:43.056294 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:43.057476 7480 hybrid_clock.cc:648] HybridClock initialized: now 1781946343057437 us; error 55 us; skew 500 ppm
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:43.060197 7480 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:05:43.061028 7480 webserver.cc:460] Webserver started at http://127.0.98.2:36791/ using document root <none> and password file <none>
I20260620 09:05:43.061646 7480 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:43.061751 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:43.065464 7480 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:05:43.067624 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.068778 7480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:43.068905 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "218fef6feaef4d58a04db592b8bf87d4"
format_stamp: "Formatted at 2026-06-20 09:05:41 on dist-test-slave-92t4"
server_key: "3c17ac5e90c47268c4583d7d379b27aa"
server_key_iv: "e309d0ae49265588a91c9d101898581e"
server_key_version: "encryptionkey@0"
I20260620 09:05:43.069519 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:43.092381 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:43.092941 6967 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
I20260620 09:05:43.095781 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:43.095932 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:43.096571 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:43.097357 7480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:43.097409 7480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.097451 7480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:43.097472 7480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.105878 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38401
I20260620 09:05:43.105907 7608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38401 every 8 connection(s)
I20260620 09:05:43.106853 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:43.116034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7480
I20260620 09:05:43.116225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6838
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:05:43.123592 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.108643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47819 (local address 127.0.98.62:35565)
0620 09:05:43.108766 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.108771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.110008 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 09:05:43.111643 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.111647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.111706 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.111764 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.113155 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.113925 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.114980 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.115275 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.119733 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.119736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.119739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.119774 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.121701 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.122237 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.122241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.122243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.122326 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.122621 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.122627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.122631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.122822 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.122899 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.123295 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.123418 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:43.124074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43077
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35565
--webserver_port=44051
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:05:43.124433 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.108641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37335 (local address 127.0.98.61:46867)
0620 09:05:43.108766 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.108771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.110142 (+ 1371us) server_negotiation.cc:407] Connection header received
0620 09:05:43.111632 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.111635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.111695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.111798 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.113120 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.113927 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115690 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.116053 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.116159 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.119114 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.119118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.119121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.119149 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.121235 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.121842 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.121846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.121848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.121949 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.122334 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.122338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.122339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.122572 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.122658 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.123968 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.124073 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:43.125036 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.109176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.98.2:37335)
0620 09:05:43.110063 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.110076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.111426 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.111958 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.111967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.112430 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.112952 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.112966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.114037 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.114040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.115057 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.115067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.116300 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.116307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.116427 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.116969 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.116992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.118953 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.121361 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.121369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.121386 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.121735 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.122056 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.122061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.122063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.122235 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.122709 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.122716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.123088 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.123220 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.123537 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":85,"spinlock_wait_cycles":39936,"thread_start_us":604,"threads_started":1}
I20260620 09:05:43.125401 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.109176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:47819)
0620 09:05:43.109643 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.109686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.111407 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.111957 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.111967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.112509 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.112975 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.112984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.114076 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.114079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.114850 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.114858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.115407 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.115598 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.117649 (+ 2051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.117669 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.119541 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.121825 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.121829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.121831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.122133 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.122441 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.122443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.122445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.122519 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.122953 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.122956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.123025 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.123751 (+ 726us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.123865 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":124,"spinlock_wait_cycles":16256,"thread_start_us":125,"threads_started":1}
I20260620 09:05:43.126621 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:43.126864 7610 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.127218 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46867
I20260620 09:05:43.127327 7611 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.127415 7610 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:43.127521 7611 heartbeater.cc:511] Master 127.0.98.61:46867 requested a full tablet report, sending...
I20260620 09:05:43.128444 6502 ts_manager.cc:194] Re-registered known tserver with Master: 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401)
I20260620 09:05:43.128578 7109 ts_manager.cc:194] Registered new tserver with Master: 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401)
I20260620 09:05:43.129225 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47819
I20260620 09:05:43.131035 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.111118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41219 (local address 127.0.98.60:42597)
0620 09:05:43.111265 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.111269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.111429 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:05:43.111928 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.111932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.111982 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.112063 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.113151 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.113678 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115174 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.115513 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.123506 (+ 7906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.123513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.123519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.123557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.125749 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.129365 (+ 3616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.129368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.129370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.129488 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.129845 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.129849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.129853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.130121 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.130219 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.130629 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.130800 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:05:43.131698 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.110690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.98.2:41219)
0620 09:05:43.111319 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.111334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.111436 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.112104 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.112109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.112509 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.112975 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.112984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.113822 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.113824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.115057 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.115067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.115625 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.115628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.115708 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.121689 (+ 5981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.121717 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.123344 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.128878 (+ 5534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.128883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.128887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.129232 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.129625 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.129631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.129636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.129732 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.130289 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.130295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.130415 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.131248 (+ 833us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.131440 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":156,"spinlock_wait_cycles":4736,"thread_start_us":390,"threads_started":1}
I20260620 09:05:43.132547 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42597
I20260620 09:05:43.132645 7609 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.132872 7609 heartbeater.cc:511] Master 127.0.98.60:42597 requested a full tablet report, sending...
I20260620 09:05:43.133670 7293 ts_manager.cc:194] Registered new tserver with Master: 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401)
W20260620 09:05:43.238381 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:43.238634 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:43.238713 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:43.242725 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:43.242797 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:43.242901 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:43.248138 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44051
--tserver_master_addrs=127.0.98.62:35565,127.0.98.61:46867,127.0.98.60:42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:43.249207 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:43.250250 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:05:43.256620 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.252805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.98.60:42597)
0620 09:05:43.252993 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.252998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.253075 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:05:43.253270 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.253275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.253341 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.253458 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.253672 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.254455 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.255440 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.256076 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.256197 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.256344 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.256385 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.256465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 09:05:43.256563 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.252704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49362)
0620 09:05:43.252948 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.252970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.253155 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.253465 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.253467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.253480 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.253570 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.253574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.254559 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.254562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.255316 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.255323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.256196 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.256198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.256261 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.256281 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.256328 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.256382 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260620 09:05:43.257530 7303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 09:05:43.257908 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9a82bd647f4e388487436440d43d76. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:43.258069 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: LEARNER
W20260620 09:05:43.261137 7624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:43.261212 7616 server_base.cc:1034] running on GCE node
W20260620 09:05:43.261132 7623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:43.261130 7626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:43.261734 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:43.262291 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:43.263471 7616 hybrid_clock.cc:648] HybridClock initialized: now 1781946343263451 us; error 29 us; skew 500 ppm
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:43.266013 7616 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:05:43.266824 7616 webserver.cc:460] Webserver started at http://127.0.98.3:44051/ using document root <none> and password file <none>
I20260620 09:05:43.267372 7616 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:43.267479 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:43.271132 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260620 09:05:43.273252 7633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.274467 7616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:43.274590 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "a60b9692ff3f4f038371a1fa72585f61"
format_stamp: "Formatted at 2026-06-20 09:05:42 on dist-test-slave-92t4"
server_key: "ae9fc909900163e642e0f921bb6c8d29"
server_key_iv: "7359c7febb070a7070dbe54c5df623ef"
server_key_version: "encryptionkey@0"
I20260620 09:05:43.275087 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:43.284773 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:43.287559 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:43.287750 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:43.288290 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:43.289196 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:43.289275 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.289340 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:43.289394 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:43.297880 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43077
I20260620 09:05:43.297945 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43077 every 8 connection(s)
I20260620 09:05:43.298842 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:43.301206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7616
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 09:05:43.314713 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.303380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.61:46867)
0620 09:05:43.303605 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.303610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.304145 (+ 535us) server_negotiation.cc:407] Connection header received
0620 09:05:43.304340 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.304345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.304411 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.304478 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.305391 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.306150 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307945 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.308649 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.308718 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.310571 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.310575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.310577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.310601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.312613 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.313242 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.313245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.313247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.313339 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.313685 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.313691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.313693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.313929 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.314065 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.314400 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.314519 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260620 09:05:43.314975 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.303086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.0.98.62:35565)
0620 09:05:43.304064 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.304068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.304082 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:43.304115 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.304117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.304157 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.304245 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.305222 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.306085 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307913 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.308255 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.308363 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.310666 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.310671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.310674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.310698 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.312615 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.313270 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.313276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.313282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.313378 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.313714 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.313718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.313720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.313937 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.314063 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.314680 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.314783 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":911,"spinlock_wait_cycles":4096,"thread_start_us":90,"threads_started":1}
Jun 20 09:05:43 dist-test-slave-92t4 krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946343, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 09:05:43.319406 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.300898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:47873)
0620 09:05:43.301813 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.301864 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.303076 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.304111 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.304119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.304526 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.304972 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.305003 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.306194 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.306199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.307217 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.307226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.307720 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.307930 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.308756 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.308781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.312421 (+ 3640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.314598 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.314605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.314621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.316063 (+ 1442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.316901 (+ 838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.316905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.316906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.317008 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.317497 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.317503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.317703 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.318151 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.318450 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":722,"threads_started":1}
I20260620 09:05:43.319736 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.300518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47873 (local address 127.0.98.62:35565)
0620 09:05:43.300678 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.300683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.301978 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 09:05:43.303343 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.303346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.303402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.303458 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.305153 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.306032 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307351 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.307581 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307688 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.312554 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.312560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.312565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.312602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.314455 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.316424 (+ 1969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.316429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.316436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.316550 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.317140 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.317143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.317145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.317352 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.317437 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.317978 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.318083 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":1920}
I20260620 09:05:43.321599 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35565
I20260620 09:05:43.318105 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:05:43.321830 7748 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.322180 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.301412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45605 (local address 127.0.98.60:42597)
0620 09:05:43.301540 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.301546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.305713 (+ 4167us) server_negotiation.cc:407] Connection header received
0620 09:05:43.305884 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.305887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.305933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.305981 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.306593 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.307348 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.308469 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.308794 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.308865 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.312601 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.312604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.312607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.312630 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.314601 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.315105 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.315108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.315110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.315188 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.315599 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.315621 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.315623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.315852 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.315920 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.321915 (+ 5995us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.322042 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:43.322371 7748 heartbeater.cc:511] Master 127.0.98.62:35565 requested a full tablet report, sending...
I20260620 09:05:43.322525 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.302375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52101 (local address 127.0.98.61:46867)
0620 09:05:43.302534 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.302538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.309756 (+ 7218us) server_negotiation.cc:407] Connection header received
0620 09:05:43.309917 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.309923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.309972 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.310029 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.310965 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.311759 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.316355 (+ 4596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.316753 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.316856 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.319091 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.319095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.319097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.319120 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.320798 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.321276 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.321281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.321284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.321365 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.321699 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.321702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.321703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.321889 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.321969 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.322282 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.322383 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 09:05:43.322644 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.301300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.98.3:45605)
0620 09:05:43.305619 (+ 4319us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.305632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.305771 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.306108 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.306110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.306318 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.306449 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.306467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.307481 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.307484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.308326 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.308338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.309009 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.309012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.309161 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.310345 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.310363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.312033 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.314708 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.314711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.314713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.315000 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.315298 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.315301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.315303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.315482 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.316003 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.316009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.321721 (+ 5712us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.322337 (+ 616us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.322471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4255,"thread_start_us":135,"threads_started":1}
W20260620 09:05:43.322815 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:43.323014 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.302272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.98.3:52101)
0620 09:05:43.309676 (+ 7404us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.309693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.309791 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.310181 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.310186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.310690 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:43.310814 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.310825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.311945 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.311948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.316242 (+ 4294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.316249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.316984 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.316991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.317110 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.317543 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:43.317556 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:43.318956 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:43.320918 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.320921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.320923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.321178 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.321490 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:43.321496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:43.321499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:43.321590 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:43.322026 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:43.322029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:43.322110 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.322677 (+ 567us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.322805 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6897,"thread_start_us":74,"threads_started":1}
I20260620 09:05:43.323413 6507 ts_manager.cc:194] Re-registered known tserver with Master: a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077)
I20260620 09:05:43.324098 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47873
I20260620 09:05:43.324173 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42597
I20260620 09:05:43.324256 7747 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.324432 7747 heartbeater.cc:511] Master 127.0.98.60:42597 requested a full tablet report, sending...
I20260620 09:05:43.324589 7749 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46867
I20260620 09:05:43.324659 7749 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:43.324795 7749 heartbeater.cc:511] Master 127.0.98.61:46867 requested a full tablet report, sending...
I20260620 09:05:43.325188 7293 ts_manager.cc:194] Registered new tserver with Master: a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077)
I20260620 09:05:43.325788 7109 ts_manager.cc:194] Registered new tserver with Master: a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077)
I20260620 09:05:43.326121 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: true } } }
I20260620 09:05:43.326241 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:43.330294 7620 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: attempting to promote NON_VOTER fa7f039bb827416a8f96302cc02164a0 to VOTER
I20260620 09:05:43.330615 7620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:43.333302 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:05:43.333624 7302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 LEARNER]: Refusing update from remote peer 7e9a82bd647f4e388487436440d43d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 09:05:43.333829 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:05:43.333979 7772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 09:05:43.335960 7765 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa7f039bb827416a8f96302cc02164a0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.336479 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa7f039bb827416a8f96302cc02164a0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.337525 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.337631 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:43.337757 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.337843 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:43.337942 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.338014 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:43.337839 7302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fa7f039bb827416a8f96302cc02164a0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.339099 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.339267 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:43.339540 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.303875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.60:42597)
0620 09:05:43.304121 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.304134 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.310648 (+ 6514us) server_negotiation.cc:407] Connection header received
0620 09:05:43.315372 (+ 4724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.315375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.315418 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.315477 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.317019 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.318060 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.319326 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.320208 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.320293 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.325094 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.325097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.325100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.325121 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.335458 (+ 10337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.337348 (+ 1890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.337352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.337354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.337455 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.337987 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.337990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.337992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.338547 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.338665 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.339029 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.339365 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260620 09:05:43.340109 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e9a82bd647f4e388487436440d43d76" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:43.340224 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:43.346197 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.340829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.0.1:49276)
0620 09:05:43.340985 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.340996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.341109 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.342167 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.342171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.342188 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.342288 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.342296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.343612 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.343615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.344596 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.344603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.345798 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.345802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.345895 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.345915 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.346005 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.346073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":3}
I20260620 09:05:43.346256 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.340961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.2:38401)
0620 09:05:43.341302 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.341314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.341337 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:05:43.341416 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.341422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.342000 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.342204 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.342379 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.343492 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.344717 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.345668 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.345801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.345957 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.346030 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.346119 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":167,"threads_started":1}
I20260620 09:05:43.346947 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.341575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43077 (local address 127.0.0.1:59968)
0620 09:05:43.342114 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.342125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.342196 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.342653 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.342656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.342672 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.342753 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.342758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.343897 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.343902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.344859 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.344869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.346568 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.346571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.346634 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.346682 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.346761 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.346821 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":19,"thread_start_us":67,"threads_started":1}
I20260620 09:05:43.347047 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.341691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.3:43077)
0620 09:05:43.341971 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.341983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.342223 (+ 240us) server_negotiation.cc:407] Connection header received
0620 09:05:43.342301 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.342307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.342522 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.342673 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.342844 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.343796 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.345108 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.346029 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.346132 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.346253 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.346811 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.346908 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260620 09:05:43.349212 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 7939f276a7c14957b71e928fcae161cb (DEFAULT_TABLE table=test-table [id=681c68c86fdd46ca9df1489e880430a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:43.350107 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.341278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:52938)
0620 09:05:43.341768 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.341779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.342279 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.343005 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.343007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.343019 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.343074 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.343077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.344690 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.344692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.346229 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.346238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.349665 (+ 3427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.349669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.349794 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.349816 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.349912 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.349976 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":103,"threads_started":1}
I20260620 09:05:43.350275 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.341432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.0.98.1:33591)
0620 09:05:43.342490 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.342499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.342516 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:05:43.342571 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.342574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.342758 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.342900 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.343202 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.344205 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.346692 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.349530 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.349633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.349805 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.349940 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.350056 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":86,"threads_started":1}
I20260620 09:05:43.350378 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7939f276a7c14957b71e928fcae161cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:43.350047 7681 tablet_service.cc:1467] Processing CreateTablet for tablet 7939f276a7c14957b71e928fcae161cb (DEFAULT_TABLE table=test-table [id=681c68c86fdd46ca9df1489e880430a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:43.351099 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7939f276a7c14957b71e928fcae161cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:43.353001 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 7939f276a7c14957b71e928fcae161cb (DEFAULT_TABLE table=test-table [id=681c68c86fdd46ca9df1489e880430a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:43.353989 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7939f276a7c14957b71e928fcae161cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:43.357503 7781 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Bootstrap starting.
I20260620 09:05:43.358596 7782 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Bootstrap starting.
I20260620 09:05:43.360986 7781 tablet_bootstrap.cc:654] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:43.361011 7782 tablet_bootstrap.cc:654] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:43.361230 7783 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Bootstrap starting.
I20260620 09:05:43.361747 7782 log.cc:824] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:43.361799 7781 log.cc:824] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:43.363251 7783 tablet_bootstrap.cc:654] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:43.363377 7782 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: No bootstrap required, opened a new log
I20260620 09:05:43.363579 7782 ts_tablet_manager.cc:1397] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:05:43.363914 7783 log.cc:824] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:43.365607 7781 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: No bootstrap required, opened a new log
I20260620 09:05:43.365762 7783 tablet_bootstrap.cc:492] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: No bootstrap required, opened a new log
I20260620 09:05:43.365788 7781 ts_tablet_manager.cc:1397] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260620 09:05:43.365919 7783 ts_tablet_manager.cc:1397] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 09:05:43.368382 7782 raft_consensus.cc:348] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.368641 7782 raft_consensus.cc:374] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:43.368692 7782 raft_consensus.cc:729] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218fef6feaef4d58a04db592b8bf87d4, State: Initialized, Role: FOLLOWER
I20260620 09:05:43.369184 7782 consensus_queue.cc:260] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.369228 7781 raft_consensus.cc:348] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.369467 7781 raft_consensus.cc:374] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:43.369521 7781 raft_consensus.cc:729] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60b9692ff3f4f038371a1fa72585f61, State: Initialized, Role: FOLLOWER
I20260620 09:05:43.369925 7781 consensus_queue.cc:260] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.370128 7782 ts_tablet_manager.cc:1428] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:05:43.370093 7783 raft_consensus.cc:348] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.370318 7783 raft_consensus.cc:374] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:43.370378 7783 raft_consensus.cc:729] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa2262f3b148959fcd48f67a39d9e5, State: Initialized, Role: FOLLOWER
I20260620 09:05:43.370815 7781 ts_tablet_manager.cc:1428] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 09:05:43.370755 7783 consensus_queue.cc:260] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.371628 7783 ts_tablet_manager.cc:1428] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:43.372572 7473 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
I20260620 09:05:43.374560 7610 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
I20260620 09:05:43.375563 7748 heartbeater.cc:503] Master 127.0.98.62:35565 was elected leader, sending a full tablet report...
W20260620 09:05:43.417433 7475 tablet.cc:2369] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:43.487742 7788 raft_consensus.cc:482] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:43.487979 7788 raft_consensus.cc:504] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.489190 7788 leader_election.cc:288] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401), 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:43.492784 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.489446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48035 (local address 127.0.98.2:38401)
0620 09:05:43.489654 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.489659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.489675 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:05:43.489733 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.489739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.489790 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.489860 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.490187 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.490792 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.491835 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.492226 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.492355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.492426 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.492592 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.492661 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 09:05:43.492795 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.489339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.98.3:48035)
0620 09:05:43.489516 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.489532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.489627 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.489899 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.489902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.489921 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.490062 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.490068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.490905 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.490911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.491707 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.491715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.492346 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.492350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.492454 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.492488 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.492576 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.492651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":11}
I20260620 09:05:43.492909 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.489663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52837 (local address 127.0.98.1:33591)
0620 09:05:43.489799 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.489803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.489814 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:43.489890 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.489893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.489942 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.490051 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:43.490342 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.490830 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.491963 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.492357 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.492492 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.492554 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.492714 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.492778 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:05:43.492926 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.489445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.98.3:52837)
0620 09:05:43.489558 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:43.489571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:43.489628 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:43.490090 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:43.490094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:43.490115 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:43.490230 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.490235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.490946 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.490950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.491844 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:43.491853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.492485 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.492489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:43.492584 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.492613 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:05:43.492700 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:05:43.492772 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":19,"spinlock_wait_cycles":4992}
I20260620 09:05:43.493567 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7939f276a7c14957b71e928fcae161cb" candidate_uuid: "a60b9692ff3f4f038371a1fa72585f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218fef6feaef4d58a04db592b8bf87d4" is_pre_election: true
I20260620 09:05:43.493829 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7939f276a7c14957b71e928fcae161cb" candidate_uuid: "a60b9692ff3f4f038371a1fa72585f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baa2262f3b148959fcd48f67a39d9e5" is_pre_election: true
I20260620 09:05:43.493880 7563 raft_consensus.cc:2454] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60b9692ff3f4f038371a1fa72585f61 in term 0.
I20260620 09:05:43.494091 7426 raft_consensus.cc:2454] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60b9692ff3f4f038371a1fa72585f61 in term 0.
I20260620 09:05:43.494405 7635 leader_election.cc:302] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218fef6feaef4d58a04db592b8bf87d4, a60b9692ff3f4f038371a1fa72585f61; no voters:
I20260620 09:05:43.494678 7788 raft_consensus.cc:2781] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:43.494751 7788 raft_consensus.cc:482] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:43.494781 7788 raft_consensus.cc:3037] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:43.495793 7788 raft_consensus.cc:504] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.496080 7788 leader_election.cc:288] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [CANDIDATE]: Term 1 election: Requested vote from peers 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2:38401), 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:43.496469 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7939f276a7c14957b71e928fcae161cb" candidate_uuid: "a60b9692ff3f4f038371a1fa72585f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baa2262f3b148959fcd48f67a39d9e5"
I20260620 09:05:43.496480 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7939f276a7c14957b71e928fcae161cb" candidate_uuid: "a60b9692ff3f4f038371a1fa72585f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218fef6feaef4d58a04db592b8bf87d4"
I20260620 09:05:43.496596 7426 raft_consensus.cc:3037] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:43.496596 7563 raft_consensus.cc:3037] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:43.497839 7563 raft_consensus.cc:2454] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60b9692ff3f4f038371a1fa72585f61 in term 1.
I20260620 09:05:43.497849 7426 raft_consensus.cc:2454] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60b9692ff3f4f038371a1fa72585f61 in term 1.
I20260620 09:05:43.498143 7635 leader_election.cc:302] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 218fef6feaef4d58a04db592b8bf87d4, 8baa2262f3b148959fcd48f67a39d9e5, a60b9692ff3f4f038371a1fa72585f61; no voters:
I20260620 09:05:43.498346 7788 raft_consensus.cc:2781] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:43.498551 7788 raft_consensus.cc:686] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 1 LEADER]: Becoming Leader. State: Replica: a60b9692ff3f4f038371a1fa72585f61, State: Running, Role: LEADER
I20260620 09:05:43.498903 7788 consensus_queue.cc:237] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } }
I20260620 09:05:43.502671 6502 catalog_manager.cc:5495] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a60b9692ff3f4f038371a1fa72585f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:43.530970 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.527373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.98.3:43077)
0620 09:05:43.527534 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.527539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.527553 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:43.527653 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.527655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.527711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.527824 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:43.528067 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.528575 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.529482 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.529753 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.529858 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.530162 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:43.530588 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:43.530725 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.530761 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.530819 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 09:05:43.537614 7426 raft_consensus.cc:1261] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Refusing update from remote peer a60b9692ff3f4f038371a1fa72585f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:43.537593 7563 raft_consensus.cc:1261] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Refusing update from remote peer a60b9692ff3f4f038371a1fa72585f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:43.538339 7790 consensus_queue.cc:1035] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:43.538576 7788 consensus_queue.cc:1035] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:43.558813 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:43.558990 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:43.563653 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e9a82bd647f4e388487436440d43d76: Sending DeleteTablet for 3 replicas of tablet 7939f276a7c14957b71e928fcae161cb
I20260620 09:05:43.564527 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 7939f276a7c14957b71e928fcae161cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52938
I20260620 09:05:43.564728 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 7939f276a7c14957b71e928fcae161cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49276
I20260620 09:05:43.565128 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet 7939f276a7c14957b71e928fcae161cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59968
I20260620 09:05:43.565605 7803 tablet_replica.cc:330] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: stopping tablet replica
I20260620 09:05:43.565943 7803 raft_consensus.cc:2229] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:43.566289 7803 raft_consensus.cc:2258] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:43.567354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6475
I20260620 09:05:43.568915 7803 ts_tablet_manager.cc:1905] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:43.572570 7803 ts_tablet_manager.cc:1918] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:43.572671 7803 log.cc:1196] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/7939f276a7c14957b71e928fcae161cb
I20260620 09:05:43.572995 7803 ts_tablet_manager.cc:1939] T 7939f276a7c14957b71e928fcae161cb P 218fef6feaef4d58a04db592b8bf87d4: Deleting consensus metadata
W20260620 09:05:43.575683 7496 connection.cc:535] server connection from 127.0.0.1:49276 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:05:43.576432 7802 tablet_replica.cc:330] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: stopping tablet replica
I20260620 09:05:43.576787 7802 raft_consensus.cc:2229] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:43.577239 7805 tablet_replica.cc:330] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: stopping tablet replica
I20260620 09:05:43.577479 7805 raft_consensus.cc:2229] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:43.577827 7805 raft_consensus.cc:2258] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:43.578197 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.566049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.0.98.61:46867)
0620 09:05:43.566148 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.566153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.566165 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:43.566200 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.566202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.566243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.566330 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.567322 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.568017 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.569514 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.569855 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.569927 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.574564 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.574570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.574576 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.574608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.576387 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.576851 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.576855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.576857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.576944 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.577278 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.577281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.577283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.577509 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.577640 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.577902 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.578016 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:05:43.580080 7802 raft_consensus.cc:2258] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:43.581866 7805 ts_tablet_manager.cc:1905] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:43.581849 7802 ts_tablet_manager.cc:1905] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:43.584422 7802 ts_tablet_manager.cc:1918] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:43.584415 7805 ts_tablet_manager.cc:1918] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:43.584508 7805 log.cc:1196] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/7939f276a7c14957b71e928fcae161cb
I20260620 09:05:43.584508 7802 log.cc:1196] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/7939f276a7c14957b71e928fcae161cb
I20260620 09:05:43.584856 7805 ts_tablet_manager.cc:1939] T 7939f276a7c14957b71e928fcae161cb P a60b9692ff3f4f038371a1fa72585f61: Deleting consensus metadata
I20260620 09:05:43.584856 7802 ts_tablet_manager.cc:1939] T 7939f276a7c14957b71e928fcae161cb P 8baa2262f3b148959fcd48f67a39d9e5: Deleting consensus metadata
W20260620 09:05:43.585870 7359 connection.cc:441] server connection from 127.0.0.1:52938 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52938 (request call id 1) could send its response
W20260620 09:05:43.585870 7636 connection.cc:441] server connection from 127.0.0.1:59968 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59968 (request call id 1) could send its response
I20260620 09:05:43.587970 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.566471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.0.98.60:42597)
0620 09:05:43.566584 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.566588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.578356 (+ 11768us) server_negotiation.cc:407] Connection header received
0620 09:05:43.578539 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.578542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.578615 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.578675 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.579626 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.580412 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.581931 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.582288 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.582386 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.584049 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.584053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.584056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.584082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.586004 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.586612 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.586615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.586617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.586694 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.586968 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.586971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.586973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.587165 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.587290 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.587574 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.587810 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:05:43.595001 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.586295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.0.98.61:46867)
0620 09:05:43.586426 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.586430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.586594 (+ 164us) server_negotiation.cc:407] Connection header received
0620 09:05:43.586765 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.586768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.586816 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.586871 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.588243 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.588833 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.589963 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.590194 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.590301 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.591657 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.591660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.591663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.591686 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.593113 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.593664 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.593667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.593669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.593750 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.594043 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.594048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.594050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.594266 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.594428 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.594662 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.594791 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:05:43.597393 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:43.586541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.0.98.60:42597)
0620 09:05:43.586647 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:43.586651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:43.589325 (+ 2674us) server_negotiation.cc:407] Connection header received
0620 09:05:43.589437 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:43.589441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:43.589482 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:43.589556 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:43.590347 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.591119 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.592276 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:43.592619 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:43.592689 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:43.594242 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:43.594246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:43.594249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:43.594271 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:43.595921 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.596286 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.596289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.596291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.596371 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:43.596605 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:43.596609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:43.596610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:43.596801 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:43.596923 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:43.597143 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 09:05:43.597247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:05:44.549587 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:44.549117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:53715)
0620 09:05:44.549374 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:44.549472 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
W20260620 09:05:44.549922 7473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:44.554662 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:44.554264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:55053)
0620 09:05:44.554483 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:44.554563 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
W20260620 09:05:44.555055 7748 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:44.555308 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:44.554928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:37617)
0620 09:05:44.555121 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:44.555221 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
W20260620 09:05:44.555576 7610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:45.064257 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e9a82bd647f4e388487436440d43d76)
I20260620 09:05:45.064287 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e9a82bd647f4e388487436440d43d76)
I20260620 09:05:45.064479 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:45.064478 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:45.065516 7815 leader_election.cc:288] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61:46867)
I20260620 09:05:45.065791 7816 leader_election.cc:288] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597)
I20260620 09:05:45.066143 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.065625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36578)
0620 09:05:45.065932 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.066038 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":56,"threads_started":1}
I20260620 09:05:45.066278 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.065630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36580)
0620 09:05:45.066094 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.066161 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":59,"threads_started":1}
W20260620 09:05:45.066339 7095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
W20260620 09:05:45.066402 7279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:45.067233 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.067000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36594)
0620 09:05:45.067115 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.067170 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:05:45.067243 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.067007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36602)
0620 09:05:45.067133 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.067179 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 09:05:45.067384 7095 leader_election.cc:334] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
W20260620 09:05:45.067485 7279 leader_election.cc:334] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:45.070817 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.065812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46867 (local address 127.0.0.1:37594)
0620 09:05:45.066372 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.066386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:45.066472 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:45.066918 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:45.066924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:45.066950 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:45.067069 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:45.067076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.068273 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.068276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:45.069525 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:45.069535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.070355 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.070358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:45.070474 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:45.070519 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 09:05:45.070595 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 09:05:45.070668 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":33,"thread_start_us":50,"threads_started":1}
I20260620 09:05:45.070823 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.066147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.98.61:46867)
0620 09:05:45.066671 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:45.066676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:45.066688 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:45.066731 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:45.066734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:45.066778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:45.067175 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:45.067274 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.068166 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.069652 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.070241 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.070337 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:45.070400 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:45.070634 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:05:45.070694 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":78,"threads_started":1}
I20260620 09:05:45.071044 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.065633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42597 (local address 127.0.0.1:49412)
0620 09:05:45.066049 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.066063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:45.066159 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:45.067225 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:45.067227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:45.067244 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:45.067358 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:45.067364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.068513 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.068516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:45.069707 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:45.069715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.070560 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.070564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:45.070654 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:45.070677 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:05:45.070748 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:05:45.070813 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":162,"thread_start_us":54,"threads_started":1}
I20260620 09:05:45.071305 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.065967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.0.98.60:42597)
0620 09:05:45.066603 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:45.066608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:45.066621 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:45.066747 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:45.066750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:45.066797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:45.067123 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:45.067713 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.068402 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.069830 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:45.070452 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:45.070527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:45.070587 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:45.070772 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 09:05:45.070848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":577,"thread_start_us":59,"threads_started":1}
I20260620 09:05:45.071301 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7f039bb827416a8f96302cc02164a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" is_pre_election: true
I20260620 09:05:45.071506 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7f039bb827416a8f96302cc02164a0 in term 1.
I20260620 09:05:45.071734 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa7f039bb827416a8f96302cc02164a0" is_pre_election: true
I20260620 09:05:45.071866 7277 leader_election.cc:302] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e854d9bf26b54fbea9b307e4dd4bce6b, fa7f039bb827416a8f96302cc02164a0; no voters: 7e9a82bd647f4e388487436440d43d76
I20260620 09:05:45.071926 7302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e854d9bf26b54fbea9b307e4dd4bce6b in term 1.
I20260620 09:05:45.072081 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:05:45.072154 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7e9a82bd647f4e388487436440d43d76)
I20260620 09:05:45.072211 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:05:45.072340 7093 leader_election.cc:302] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e854d9bf26b54fbea9b307e4dd4bce6b, fa7f039bb827416a8f96302cc02164a0; no voters: 7e9a82bd647f4e388487436440d43d76
I20260620 09:05:45.072561 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 09:05:45.072628 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7e9a82bd647f4e388487436440d43d76)
I20260620 09:05:45.072685 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:05:45.073467 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:45.073836 7815 leader_election.cc:288] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 election: Requested vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), e854d9bf26b54fbea9b307e4dd4bce6b (127.0.98.61:46867)
I20260620 09:05:45.073855 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:45.074131 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7f039bb827416a8f96302cc02164a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b"
I20260620 09:05:45.074223 7816 leader_election.cc:288] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 election: Requested vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597)
I20260620 09:05:45.074321 7119 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fa7f039bb827416a8f96302cc02164a0 in current term 2: Already voted for candidate e854d9bf26b54fbea9b307e4dd4bce6b in this term.
I20260620 09:05:45.074512 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa7f039bb827416a8f96302cc02164a0"
I20260620 09:05:45.074669 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.074284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36610)
0620 09:05:45.074482 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.074538 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 09:05:45.074668 7302 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e854d9bf26b54fbea9b307e4dd4bce6b in current term 2: Already voted for candidate fa7f039bb827416a8f96302cc02164a0 in this term.
I20260620 09:05:45.074681 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.073952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36604)
0620 09:05:45.074040 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.074624 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 09:05:45.075330 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.075137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36618)
0620 09:05:45.075234 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.075278 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 09:05:45.075340 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.075146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:36630)
0620 09:05:45.075244 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.075287 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 09:05:45.075428 7279 leader_election.cc:334] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
W20260620 09:05:45.075445 7095 leader_election.cc:334] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:45.075486 7279 leader_election.cc:302] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa7f039bb827416a8f96302cc02164a0; no voters: 7e9a82bd647f4e388487436440d43d76, e854d9bf26b54fbea9b307e4dd4bce6b
I20260620 09:05:45.075498 7095 leader_election.cc:302] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e854d9bf26b54fbea9b307e4dd4bce6b; no voters: 7e9a82bd647f4e388487436440d43d76, fa7f039bb827416a8f96302cc02164a0
I20260620 09:05:45.075621 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:05:45.075736 7815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 09:05:45.550953 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.550470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:38887)
0620 09:05:45.550762 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.550835 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260620 09:05:45.555891 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.555540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:58871)
0620 09:05:45.555754 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.555808 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260620 09:05:45.556401 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:45.556051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:46403)
0620 09:05:45.556282 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:45.556327 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260620 09:05:46.552596 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.551764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:38073)
0620 09:05:46.552411 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.552486 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":80,"threads_started":1}
I20260620 09:05:46.556877 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.556468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:58815)
0620 09:05:46.556687 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.556742 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260620 09:05:46.557354 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.556969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:37821)
0620 09:05:46.557236 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.557281 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260620 09:05:46.906540 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:46.906677 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:46.907042 7831 leader_election.cc:288] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597)
I20260620 09:05:46.907503 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa7f039bb827416a8f96302cc02164a0" is_pre_election: true
I20260620 09:05:46.907665 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.907250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55136)
0620 09:05:46.907497 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.907556 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260620 09:05:46.907658 7302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e854d9bf26b54fbea9b307e4dd4bce6b in term 2.
I20260620 09:05:46.907975 7093 leader_election.cc:302] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e854d9bf26b54fbea9b307e4dd4bce6b, fa7f039bb827416a8f96302cc02164a0; no voters:
I20260620 09:05:46.908170 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 09:05:46.908227 7831 raft_consensus.cc:482] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:46.908254 7831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:05:46.908524 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.908330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55144)
0620 09:05:46.908419 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.908464 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 09:05:46.908634 7095 leader_election.cc:334] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:46.909428 7831 raft_consensus.cc:504] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:46.909758 7831 leader_election.cc:288] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 election: Requested vote from peers 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565), fa7f039bb827416a8f96302cc02164a0 (127.0.98.60:42597)
I20260620 09:05:46.910082 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.909849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55154)
0620 09:05:46.909957 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.910006 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 09:05:46.910199 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa7f039bb827416a8f96302cc02164a0"
I20260620 09:05:46.910343 7302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:05:46.910730 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.910529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55158)
0620 09:05:46.910620 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.910679 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:05:46.910890 7095 leader_election.cc:334] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
I20260620 09:05:46.911468 7302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e854d9bf26b54fbea9b307e4dd4bce6b in term 3.
I20260620 09:05:46.911748 7093 leader_election.cc:302] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e854d9bf26b54fbea9b307e4dd4bce6b, fa7f039bb827416a8f96302cc02164a0; no voters: 7e9a82bd647f4e388487436440d43d76
I20260620 09:05:46.911888 7831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:05:46.912037 7831 raft_consensus.cc:686] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [term 3 LEADER]: Becoming Leader. State: Replica: e854d9bf26b54fbea9b307e4dd4bce6b, State: Running, Role: LEADER
I20260620 09:05:46.912302 7831 consensus_queue.cc:237] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } }
I20260620 09:05:46.913319 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e854d9bf26b54fbea9b307e4dd4bce6b. Latest consensus state: current_term: 3 leader_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:46.913427 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:46.913723 7836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:46.915839 7836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:46.916182 7836 catalog_manager.cc:1259] Loaded cluster ID: 8b17336f40834446aad6fd299496890e
I20260620 09:05:46.916222 7836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:46.916735 7836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:46.917094 7836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Loaded TSK: 0
I20260620 09:05:46.917709 7836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:05:46.969575 7109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:46.972682 7109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:46.975821 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.975547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55172)
0620 09:05:46.975683 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.975741 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 09:05:46.975837 7302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [term 3 FOLLOWER]: Refusing update from remote peer e854d9bf26b54fbea9b307e4dd4bce6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 09:05:46.976287 7831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 09:05:46.976575 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.976319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.0.1:55188)
0620 09:05:46.976469 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.976525 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 09:05:46.976749 7095 consensus_peers.cc:487] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b -> Peer 7e9a82bd647f4e388487436440d43d76 (127.0.98.62:35565): Couldn't send request to peer 7e9a82bd647f4e388487436440d43d76. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 09:05:46.977916 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e854d9bf26b54fbea9b307e4dd4bce6b. Latest consensus state: current_term: 3 leader_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:46.978072 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:46.979192 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:46.979298 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:46.981280 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7e9a82bd647f4e388487436440d43d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35565 } } peers { permanent_uuid: "e854d9bf26b54fbea9b307e4dd4bce6b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46867 } attrs { promote: false } } peers { permanent_uuid: "fa7f039bb827416a8f96302cc02164a0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42597 } attrs { promote: false } } }
I20260620 09:05:46.981389 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7f039bb827416a8f96302cc02164a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:05:46.989099 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.984772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43077 (local address 127.0.0.1:52270)
0620 09:05:46.984886 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.984903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:46.984997 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:46.985533 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:46.985536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:46.985554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:46.985697 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.985704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.986840 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.986843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.987783 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.987791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.988683 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.988687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.988809 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.988835 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 09:05:46.988879 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 09:05:46.988951 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":10}
I20260620 09:05:46.989684 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.984899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.0.98.3:43077)
0620 09:05:46.985272 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:46.985277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:46.985292 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:05:46.985346 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:46.985349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:46.985401 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:46.985823 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:46.985972 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.986714 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.987940 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.988560 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.989353 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.989411 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:46.989448 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 09:05:46.989516 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":63,"threads_started":1}
I20260620 09:05:46.989755 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.984985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:42372)
0620 09:05:46.985727 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.985742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:46.985824 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:46.986175 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:46.986178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:46.986194 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:46.986286 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.986292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.987273 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.987277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.988254 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.988261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.989363 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.989366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.989472 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.989489 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:05:46.989578 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:05:46.989643 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":102,"threads_started":1}
I20260620 09:05:46.990108 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.985357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.0.98.1:33591)
0620 09:05:46.985632 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:46.985637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:46.985899 (+ 262us) server_negotiation.cc:407] Connection header received
0620 09:05:46.985958 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:46.985961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:46.986005 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:46.986079 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:46.986425 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.987155 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.988376 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.989094 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.989250 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.989309 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:46.989915 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 09:05:46.989990 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
I20260620 09:05:46.990249 7681 tablet_service.cc:1467] Processing CreateTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 (DEFAULT_TABLE table=test-table [id=cdd01a65e32d43e49cc87099eb8e7eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:46.990511 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.985373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.0.1:34768)
0620 09:05:46.986363 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.986376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:46.986531 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:46.986787 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:46.986789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:46.986804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:46.986949 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.986954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.987935 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.987938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.989052 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:46.989060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.990069 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.990073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:46.990161 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.990210 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 09:05:46.990276 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:05:46.990343 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":353,"thread_start_us":73,"threads_started":1}
I20260620 09:05:46.990554 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d233f1abf47e296cbedd9b3f12c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:46.990697 7406 tablet_service.cc:1467] Processing CreateTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 (DEFAULT_TABLE table=test-table [id=cdd01a65e32d43e49cc87099eb8e7eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:46.990864 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.985509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.98.2:38401)
0620 09:05:46.985757 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:46.985764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:46.986480 (+ 716us) server_negotiation.cc:407] Connection header received
0620 09:05:46.986633 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:46.986635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:46.986687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:46.986771 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:46.987049 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.987809 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.989211 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:46.989843 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:46.989944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:46.990011 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:46.990655 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 09:05:46.990747 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260620 09:05:46.990979 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d233f1abf47e296cbedd9b3f12c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:46.992854 7849 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Bootstrap starting.
I20260620 09:05:46.992920 7850 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: Bootstrap starting.
I20260620 09:05:46.993508 7543 tablet_service.cc:1467] Processing CreateTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 (DEFAULT_TABLE table=test-table [id=cdd01a65e32d43e49cc87099eb8e7eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:46.993769 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71d233f1abf47e296cbedd9b3f12c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:46.993810 7850 tablet_bootstrap.cc:654] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:46.993806 7849 tablet_bootstrap.cc:654] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:46.994876 7849 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: No bootstrap required, opened a new log
I20260620 09:05:46.994974 7849 ts_tablet_manager.cc:1397] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:46.995306 7849 raft_consensus.cc:348] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.995414 7849 raft_consensus.cc:374] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:46.995457 7849 raft_consensus.cc:729] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60b9692ff3f4f038371a1fa72585f61, State: Initialized, Role: FOLLOWER
I20260620 09:05:46.995572 7849 consensus_queue.cc:260] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.995800 7853 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: Bootstrap starting.
I20260620 09:05:46.995977 7849 ts_tablet_manager.cc:1428] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:46.996598 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.996346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:57309)
0620 09:05:46.996468 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.996531 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:05:46.997385 7853 tablet_bootstrap.cc:654] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:46.997646 7850 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: No bootstrap required, opened a new log
I20260620 09:05:46.997781 7850 ts_tablet_manager.cc:1397] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 09:05:46.998085 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.997884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:56877)
0620 09:05:46.997975 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.998025 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:05:46.998391 7850 raft_consensus.cc:348] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.998550 7850 raft_consensus.cc:374] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:46.998598 7850 raft_consensus.cc:729] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa2262f3b148959fcd48f67a39d9e5, State: Initialized, Role: FOLLOWER
I20260620 09:05:46.998732 7850 consensus_queue.cc:260] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.998951 7853 tablet_bootstrap.cc:492] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: No bootstrap required, opened a new log
I20260620 09:05:46.999042 7853 ts_tablet_manager.cc:1397] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:05:46.999161 7850 ts_tablet_manager.cc:1428] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:46.999425 7853 raft_consensus.cc:348] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.999528 7853 raft_consensus.cc:374] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:46.999559 7853 raft_consensus.cc:729] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218fef6feaef4d58a04db592b8bf87d4, State: Initialized, Role: FOLLOWER
I20260620 09:05:46.999727 7853 consensus_queue.cc:260] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:46.999941 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:46.999696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:46831)
0620 09:05:46.999828 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:46.999878 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 09:05:47.000078 7853 ts_tablet_manager.cc:1428] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:47.001044 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.000786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:47195)
0620 09:05:47.000927 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.000976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 09:05:47.001487 7474 heartbeater.cc:503] Master 127.0.98.61:46867 was elected leader, sending a full tablet report...
I20260620 09:05:47.001781 7611 heartbeater.cc:503] Master 127.0.98.61:46867 was elected leader, sending a full tablet report...
I20260620 09:05:47.002260 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.001996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:58301)
0620 09:05:47.002131 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.002185 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 09:05:47.002763 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.002527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:46091)
0620 09:05:47.002662 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.002704 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 09:05:47.032354 7857 raft_consensus.cc:482] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:47.032505 7857 raft_consensus.cc:504] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:47.033535 7857 leader_election.cc:288] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077), 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:47.037485 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.033793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43077 (local address 127.0.98.2:52287)
0620 09:05:47.033930 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.033948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:47.034076 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:47.034400 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:47.034403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:47.034421 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:47.034538 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:47.034543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.035268 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.035274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:47.036080 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:47.036088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.037021 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.037025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:47.037149 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:47.037179 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:05:47.037259 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:05:47.037341 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 09:05:47.037506 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.033907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52287 (local address 127.0.98.3:43077)
0620 09:05:47.034049 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:47.034054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:47.034070 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:05:47.034177 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:47.034183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:47.034240 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:47.034337 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:47.034651 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.035155 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.036201 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.036853 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.036983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:47.037072 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:47.037285 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:05:47.037375 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:05:47.037940 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.034054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.98.2:60661)
0620 09:05:47.034394 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.034408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:47.034504 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:47.034753 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:47.034756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:47.034771 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:47.034889 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:47.034894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.035592 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.035598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:47.036495 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:47.036524 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.037427 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.037431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:47.037591 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:47.037618 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:05:47.037738 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:05:47.037807 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260620 09:05:47.037938 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.034183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60661 (local address 127.0.98.1:33591)
0620 09:05:47.034397 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:47.034403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:47.034465 (+ 62us) server_negotiation.cc:407] Connection header received
0620 09:05:47.034604 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:47.034607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:47.034657 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:47.034734 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:47.034994 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.035484 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.036641 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.037271 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.037432 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:47.037487 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:47.037757 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 09:05:47.037834 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 09:05:47.038002 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d233f1abf47e296cbedd9b3f12c03" candidate_uuid: "218fef6feaef4d58a04db592b8bf87d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60b9692ff3f4f038371a1fa72585f61" is_pre_election: true
I20260620 09:05:47.038283 7701 raft_consensus.cc:2454] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218fef6feaef4d58a04db592b8bf87d4 in term 0.
I20260620 09:05:47.038456 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d233f1abf47e296cbedd9b3f12c03" candidate_uuid: "218fef6feaef4d58a04db592b8bf87d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baa2262f3b148959fcd48f67a39d9e5" is_pre_election: true
I20260620 09:05:47.038623 7426 raft_consensus.cc:2454] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218fef6feaef4d58a04db592b8bf87d4 in term 0.
I20260620 09:05:47.038777 7498 leader_election.cc:302] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218fef6feaef4d58a04db592b8bf87d4, a60b9692ff3f4f038371a1fa72585f61; no voters:
I20260620 09:05:47.039064 7857 raft_consensus.cc:2781] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:47.039119 7857 raft_consensus.cc:482] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:47.039146 7857 raft_consensus.cc:3037] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:47.040175 7857 raft_consensus.cc:504] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:47.040519 7857 leader_election.cc:288] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [CANDIDATE]: Term 1 election: Requested vote from peers a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077), 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591)
I20260620 09:05:47.040854 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d233f1abf47e296cbedd9b3f12c03" candidate_uuid: "218fef6feaef4d58a04db592b8bf87d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60b9692ff3f4f038371a1fa72585f61"
I20260620 09:05:47.040975 7701 raft_consensus.cc:3037] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:47.041018 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71d233f1abf47e296cbedd9b3f12c03" candidate_uuid: "218fef6feaef4d58a04db592b8bf87d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baa2262f3b148959fcd48f67a39d9e5"
I20260620 09:05:47.041162 7426 raft_consensus.cc:3037] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:47.041905 7701 raft_consensus.cc:2454] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218fef6feaef4d58a04db592b8bf87d4 in term 1.
I20260620 09:05:47.042098 7426 raft_consensus.cc:2454] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218fef6feaef4d58a04db592b8bf87d4 in term 1.
I20260620 09:05:47.042174 7498 leader_election.cc:302] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218fef6feaef4d58a04db592b8bf87d4, a60b9692ff3f4f038371a1fa72585f61; no voters:
I20260620 09:05:47.042341 7857 raft_consensus.cc:2781] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:47.042569 7857 raft_consensus.cc:686] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 1 LEADER]: Becoming Leader. State: Replica: 218fef6feaef4d58a04db592b8bf87d4, State: Running, Role: LEADER
I20260620 09:05:47.042832 7857 consensus_queue.cc:237] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260620 09:05:47.043228 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.042978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:55709)
0620 09:05:47.043117 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.043166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 09:05:47.045529 7108 catalog_manager.cc:5495] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218fef6feaef4d58a04db592b8bf87d4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "218fef6feaef4d58a04db592b8bf87d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218fef6feaef4d58a04db592b8bf87d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: HEALTHY } } }
W20260620 09:05:47.053514 7750 tablet.cc:2369] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:47.056932 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.053888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.0.98.2:38401)
0620 09:05:47.054009 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:47.054013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:47.054025 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:47.054088 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:47.054090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:47.054141 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:47.054226 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:47.054429 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.054904 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.055659 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:47.055861 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:47.055932 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:47.056139 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:47.056583 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:47.056668 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:47.056720 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:05:47.056765 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:47.059885 7426 raft_consensus.cc:1261] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Refusing update from remote peer 218fef6feaef4d58a04db592b8bf87d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:47.060120 7701 raft_consensus.cc:1261] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Refusing update from remote peer 218fef6feaef4d58a04db592b8bf87d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:47.060434 7859 consensus_queue.cc:1035] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8baa2262f3b148959fcd48f67a39d9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:47.060658 7857 consensus_queue.cc:1035] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60b9692ff3f4f038371a1fa72585f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:47.062338 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.062044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:41895)
0620 09:05:47.062189 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.062247 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 09:05:47.064287 7749 heartbeater.cc:503] Master 127.0.98.61:46867 was elected leader, sending a full tablet report...
I20260620 09:05:47.064577 7862 mvcc.cc:205] Tried to move back new op lower bound from 7298852237552320512 to 7298852237490274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298852237552320512}]
I20260620 09:05:47.065101 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.064439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:40117)
0620 09:05:47.064939 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.064995 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
I20260620 09:05:47.065438 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.064966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:57457)
0620 09:05:47.065292 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.065361 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260620 09:05:47.066766 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.066491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.1:44999)
0620 09:05:47.066651 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.066700 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 09:05:47.067770 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.067558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.2:39257)
0620 09:05:47.067661 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.067710 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 09:05:47.067793 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.067595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35565 (local address 127.0.98.3:32897)
0620 09:05:47.067684 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.067730 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:05:47.068971 7865 mvcc.cc:205] Tried to move back new op lower bound from 7298852237552320512 to 7298852237490274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298852237552320512}]
I20260620 09:05:47.073652 7108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:47.073808 7108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:47.077523 7108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e854d9bf26b54fbea9b307e4dd4bce6b: Sending DeleteTablet for 3 replicas of tablet a71d233f1abf47e296cbedd9b3f12c03
I20260620 09:05:47.078326 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:47 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52270
I20260620 09:05:47.078470 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:47 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42372
I20260620 09:05:47.078656 7872 tablet_replica.cc:330] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: stopping tablet replica
I20260620 09:05:47.078794 7872 raft_consensus.cc:2229] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:47.078843 7873 tablet_replica.cc:330] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5: stopping tablet replica
I20260620 09:05:47.078963 7872 raft_consensus.cc:2258] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:47.079047 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet a71d233f1abf47e296cbedd9b3f12c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:47 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34768
I20260620 09:05:47.078967 7873 raft_consensus.cc:2229] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:47.079468 7873 raft_consensus.cc:2258] T a71d233f1abf47e296cbedd9b3f12c03 P 8baa2262f3b148959fcd48f67a39d9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:47.079681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7253
I20260620 09:05:47.084048 7874 tablet_replica.cc:330] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: stopping tablet replica
I20260620 09:05:47.084175 7874 raft_consensus.cc:2229] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:47.084487 7874 raft_consensus.cc:2258] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:47.085426 7874 ts_tablet_manager.cc:1905] T a71d233f1abf47e296cbedd9b3f12c03 P 218fef6feaef4d58a04db592b8bf87d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:47.085888 7872 ts_tablet_manager.cc:1905] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 09:05:47.087400 7848 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:47.086267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:42386)
0620 09:05:47.086393 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.086410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:47.086488 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:47.087286 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:05:47.087585 7095 catalog_manager.cc:4584] TS 8baa2262f3b148959fcd48f67a39d9e5 (127.0.98.1:33591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a71d233f1abf47e296cbedd9b3f12c03: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:05:47.088119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7480
I20260620 09:05:47.090276 7872 ts_tablet_manager.cc:1918] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:47.090354 7872 log.cc:1196] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/a71d233f1abf47e296cbedd9b3f12c03
I20260620 09:05:47.090636 7872 ts_tablet_manager.cc:1939] T a71d233f1abf47e296cbedd9b3f12c03 P a60b9692ff3f4f038371a1fa72585f61: Deleting consensus metadata
I20260620 09:05:47.091291 7095 catalog_manager.cc:4841] TS a60b9692ff3f4f038371a1fa72585f61 (127.0.98.3:43077): tablet a71d233f1abf47e296cbedd9b3f12c03 (table test-table [id=cdd01a65e32d43e49cc87099eb8e7eda]) successfully deleted
I20260620 09:05:47.094005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7616
I20260620 09:05:47.094218 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:47.093955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.0.1:34782)
0620 09:05:47.094048 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:47.094128 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 09:05:47.100427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 6972
I20260620 09:05:47.106026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7081
2026-06-20T09:05:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7571 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:47 dist-test-slave-92t4 krb5kdc[7877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:47 dist-test-slave-92t4 krb5kdc[7877](info): set up 2 sockets
Jun 20 09:05:47 dist-test-slave-92t4 krb5kdc[7877](info): commencing operation
krb5kdc: starting...
W20260620 09:05:48.149649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 09:05:48.267596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35715
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:48.375339 7890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:48.375613 7890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:48.375672 7890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:48.379681 7890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:48.379746 7890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 09:05:48.379763 7890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:48.379781 7890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:48.379796 7890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:48.379812 7890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:48.385278 7890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:48.386749 7890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:48.387861 7890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:48.396766 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:48.396778 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:48.396930 7890 server_base.cc:1034] running on GCE node
W20260620 09:05:48.397105 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:48.397710 7890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:48.398969 7890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:48.400173 7890 hybrid_clock.cc:648] HybridClock initialized: now 1781946348400157 us; error 39 us; skew 500 ppm
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:48.403373 7890 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 09:05:48.404325 7890 webserver.cc:460] Webserver started at http://127.0.98.62:39671/ using document root <none> and password file <none>
I20260620 09:05:48.404922 7890 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:48.404989 7890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:48.405261 7890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:48.407553 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "e69aa97e99b043628b8f38d859d1b1aa"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "fd4f2579676d2952fd39c06f94a3fe8c"
server_key_iv: "d5cfc3eee2cb1db912f0ff8c0e755049"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.408128 7890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "e69aa97e99b043628b8f38d859d1b1aa"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "fd4f2579676d2952fd39c06f94a3fe8c"
server_key_iv: "d5cfc3eee2cb1db912f0ff8c0e755049"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.412185 7890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:05:48.414849 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:48.416229 7890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:48.416342 7890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "e69aa97e99b043628b8f38d859d1b1aa"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "fd4f2579676d2952fd39c06f94a3fe8c"
server_key_iv: "d5cfc3eee2cb1db912f0ff8c0e755049"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.416443 7890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:48.441812 7890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:48.445519 7890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:48.445709 7890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:48.454229 7890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35715
I20260620 09:05:48.454233 7967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35715 every 8 connection(s)
I20260620 09:05:48.455366 7890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:48.458374 7968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:48.463990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7890
I20260620 09:05:48.464125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:48.464233 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Bootstrap starting.
I20260620 09:05:48.464421 392 external_mini_cluster.cc:1428] Setting key d7650f534d470378d713ea45be89d4a6
I20260620 09:05:48.467002 7968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:48.467792 7968 log.cc:824] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:48.469990 7968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: No bootstrap required, opened a new log
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:48.474566 7968 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } }
I20260620 09:05:48.474840 7968 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:48.474923 7968 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69aa97e99b043628b8f38d859d1b1aa, State: Initialized, Role: FOLLOWER
I20260620 09:05:48.475471 7968 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } }
I20260620 09:05:48.475633 7968 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:48.475706 7968 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:48.475826 7968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:48.477095 7968 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } }
I20260620 09:05:48.477494 7968 leader_election.cc:302] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69aa97e99b043628b8f38d859d1b1aa; no voters:
I20260620 09:05:48.477869 7968 leader_election.cc:288] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:48.477962 7973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:48.478152 7973 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [term 1 LEADER]: Becoming Leader. State: Replica: e69aa97e99b043628b8f38d859d1b1aa, State: Running, Role: LEADER
I20260620 09:05:48.478480 7973 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } }
I20260620 09:05:48.479257 7968 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:48.480484 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69aa97e99b043628b8f38d859d1b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } } }
I20260620 09:05:48.480615 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:48.480481 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69aa97e99b043628b8f38d859d1b1aa. Latest consensus state: current_term: 1 leader_uuid: "e69aa97e99b043628b8f38d859d1b1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69aa97e99b043628b8f38d859d1b1aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35715 } } }
I20260620 09:05:48.480903 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:48.480939 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.465876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.62:35715)
0620 09:05:48.466347 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:48.466374 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:48.466412 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:05:48.467189 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:48.467211 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:48.467600 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:48.467945 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:48.468802 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.469961 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.471141 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.471672 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.471812 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.474679 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:48.474685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:48.474700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:48.474730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:48.476937 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.477512 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.477520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.477527 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.477669 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.478034 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.478040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.478042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.478660 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:48.478880 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:48.479241 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 09:05:48.479726 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
I20260620 09:05:48.481208 7982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:48.484577 7982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:48.490852 7982 catalog_manager.cc:1347] Generated new cluster ID: 505e341873994836997e42ed5b114b22
I20260620 09:05:48.490947 7982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:48.499688 7982 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:48.500679 7982 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:48.508554 7982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Generated new TSK 0
I20260620 09:05:48.509377 7982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 09:05:48.540920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35715
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:48.655463 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:48.655804 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:48.655912 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:48.660048 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:48.660147 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:48.660238 7994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:48.660327 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:48.665709 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:48.666929 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:48.668035 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:48.677605 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:48.677672 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:48.677800 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:48.677961 7994 server_base.cc:1034] running on GCE node
I20260620 09:05:48.678411 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:48.679039 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:48.680219 7994 hybrid_clock.cc:648] HybridClock initialized: now 1781946348680191 us; error 41 us; skew 500 ppm
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:48.682992 7994 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:05:48.683876 7994 webserver.cc:460] Webserver started at http://127.0.98.1:45623/ using document root <none> and password file <none>
I20260620 09:05:48.684398 7994 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:48.684456 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:48.684620 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:48.686751 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "94a345ba5aa74973a8b24af3fbf2b427"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "d898d6d36e9ad77beb37ac95f62345ed"
server_key_iv: "aefa8e26e3b2a6100a89e091a28d7302"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.687353 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94a345ba5aa74973a8b24af3fbf2b427"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "d898d6d36e9ad77beb37ac95f62345ed"
server_key_iv: "aefa8e26e3b2a6100a89e091a28d7302"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.691097 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:05:48.693434 8009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:48.694643 7994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:05:48.694737 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "94a345ba5aa74973a8b24af3fbf2b427"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "d898d6d36e9ad77beb37ac95f62345ed"
server_key_iv: "aefa8e26e3b2a6100a89e091a28d7302"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.694819 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:48.704583 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:48.707893 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:48.708041 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:48.709475 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:48.709527 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:48.709568 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:48.709627 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:48.720319 7994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40529
I20260620 09:05:48.720405 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40529 every 8 connection(s)
I20260620 09:05:48.721315 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:48.722986 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.710237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56869 (local address 127.0.98.62:35715)
0620 09:05:48.710381 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:48.710386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:48.711276 (+ 890us) server_negotiation.cc:407] Connection header received
0620 09:05:48.712274 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:48.712280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:48.712345 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:48.712414 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:48.713986 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.714625 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.715768 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.716027 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.716146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.719346 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:48.719349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:48.719351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:48.719376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:48.720935 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.721532 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.721535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.721536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.721612 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.721998 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.722000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.722002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.722219 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:48.722319 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:48.722766 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 09:05:48.722867 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:05:48.723749 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.710640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.1:56869)
0620 09:05:48.711093 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:48.711144 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:48.712006 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:48.712839 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:48.712846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:48.713277 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:48.713839 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.713852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.714754 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.714758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.715643 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.715651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.716198 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.716202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.716369 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.717016 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:48.717031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:48.719158 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:48.721073 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.721078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.721092 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.721436 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.721714 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.721719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.721721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.721886 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.722331 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:48.722337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:48.722554 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 09:05:48.722703 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 09:05:48.722955 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
I20260620 09:05:48.724898 8125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35715
I20260620 09:05:48.725136 8125 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:48.725636 8125 heartbeater.cc:511] Master 127.0.98.62:35715 requested a full tablet report, sending...
I20260620 09:05:48.727171 7922 ts_manager.cc:194] Registered new tserver with Master: 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529)
I20260620 09:05:48.727667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 7994
I20260620 09:05:48.727797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:48.728106 392 external_mini_cluster.cc:1428] Setting key f2b2fcf944b0fd51c11d86bfdc096fc7
I20260620 09:05:48.728689 7922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56869
I20260620 09:05:48.734953 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.727229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36275 (local address 127.0.98.62:35715)
0620 09:05:48.727368 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:48.727372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:48.727565 (+ 193us) server_negotiation.cc:407] Connection header received
0620 09:05:48.727687 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:48.727689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:48.727761 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:48.727836 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:48.728374 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.729103 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.730141 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.730433 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.730537 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.731793 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:48.731799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:48.731803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:48.731832 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:48.733334 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.733825 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.733828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.733829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.733897 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.734206 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.734209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.734211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.734377 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:48.734455 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:48.734744 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 09:05:48.734832 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 09:05:48.735155 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.727148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.1:36275)
0620 09:05:48.727483 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:48.727494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:48.727600 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:48.727954 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:48.727956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:48.728122 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:48.728266 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.728272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.729233 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.729237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.730011 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.730023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.730561 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.730567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.730696 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.731275 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:48.731295 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:48.731636 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:48.733461 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.733467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.733472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.733733 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.733990 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.733996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.734001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.734105 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.734475 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:48.734479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:48.734585 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 09:05:48.734699 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:05:48.734878 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 09:05:48.754021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35715
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:48.865427 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:48.865731 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:48.865816 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:48.869900 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:48.869987 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:48.870056 8134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:48.870142 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:48.875440 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:48.876523 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:48.877589 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:48.887727 8142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:48.887732 8139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:48.887746 8140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:48.887998 8134 server_base.cc:1034] running on GCE node
I20260620 09:05:48.888406 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:48.889103 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:48.890319 8134 hybrid_clock.cc:648] HybridClock initialized: now 1781946348890282 us; error 31 us; skew 500 ppm
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:48.892980 8134 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:05:48.893922 8134 webserver.cc:460] Webserver started at http://127.0.98.2:34237/ using document root <none> and password file <none>
I20260620 09:05:48.894448 8134 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:48.894507 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:48.894672 8134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:48.896641 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "265f21c300f64f638d5a4508ede55e6d"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "3400d285e19b223d5bc1d8ad93caa8b7"
server_key_iv: "3d2a91ac45902fad17d972fb32776815"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.897217 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "265f21c300f64f638d5a4508ede55e6d"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "3400d285e19b223d5bc1d8ad93caa8b7"
server_key_iv: "3d2a91ac45902fad17d972fb32776815"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.901243 8134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:05:48.903616 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:48.904826 8134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:48.904930 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "265f21c300f64f638d5a4508ede55e6d"
format_stamp: "Formatted at 2026-06-20 09:05:48 on dist-test-slave-92t4"
server_key: "3400d285e19b223d5bc1d8ad93caa8b7"
server_key_iv: "3d2a91ac45902fad17d972fb32776815"
server_key_version: "encryptionkey@0"
I20260620 09:05:48.905016 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:48.922935 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:48.926029 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:48.926194 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:48.927675 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:48.927729 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:48.927771 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:48.927824 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:48 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:48.938462 8134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36115
I20260620 09:05:48.939005 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36115 every 8 connection(s)
I20260620 09:05:48.939657 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:48.939888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8134
I20260620 09:05:48.940026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:48.940361 392 external_mini_cluster.cc:1428] Setting key 1e2af8afcbb1081771ebf287b9e0829d
I20260620 09:05:48.941969 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.928396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39551 (local address 127.0.98.62:35715)
0620 09:05:48.928549 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:48.928554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:48.929494 (+ 940us) server_negotiation.cc:407] Connection header received
0620 09:05:48.930593 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:48.930597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:48.930674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:48.930738 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:48.932083 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.932704 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.933950 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.934185 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.934300 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.937732 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:48.937735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:48.937737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:48.937761 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:48.939460 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.940103 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.940109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.940114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.940208 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.940679 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.940682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.940684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.940984 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:48.941123 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:48.941679 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 09:05:48.941809 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:05:48.942778 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.928841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.2:39551)
0620 09:05:48.929330 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:48.929381 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:48.930342 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:48.931082 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:48.931089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:48.931477 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:48.931943 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.931956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.932832 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.932836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.933825 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.933836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.934358 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.934363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.934519 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.935131 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:48.935156 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:48.937557 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:48.939566 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.939570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.939581 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.939985 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.940379 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.940384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.940388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.940555 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.941132 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:48.941138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:48.941409 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 09:05:48.941560 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 09:05:48.941902 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":164,"threads_started":1}
I20260620 09:05:48.944200 8265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35715
I20260620 09:05:48.944355 8265 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:48.944923 8265 heartbeater.cc:511] Master 127.0.98.62:35715 requested a full tablet report, sending...
I20260620 09:05:48.945983 7922 ts_manager.cc:194] Registered new tserver with Master: 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:48.946666 7922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39551
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:05:48.953770 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.946638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55531 (local address 127.0.98.62:35715)
0620 09:05:48.946734 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:48.946740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:48.946755 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:05:48.946791 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:48.946794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:48.946841 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:48.946950 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:48.947490 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.948164 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.949181 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.949429 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.949540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.950728 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:48.950732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:48.950735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:48.950763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:48.952152 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.952610 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.952615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.952617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.952694 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:48.952976 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:48.952981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:48.952983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:48.953236 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:48.953353 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:48.953567 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 09:05:48.953658 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 09:05:48.953783 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:48.946302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.2:55531)
0620 09:05:48.946601 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:48.946616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:48.946703 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:48.946983 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:48.946986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:48.947204 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:48.947371 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.947377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.948281 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.948284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.948847 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:48.948855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:48.949555 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:48.949559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:48.949660 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:48.950251 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:48.950263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:48.950588 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:48.952260 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.952264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.952266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.952508 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.952784 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:48.952789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:48.952791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:48.952880 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:48.953353 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:48.953355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:48.953434 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 09:05:48.953526 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:05:48.953636 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:05:48.970256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35715
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:49.083127 8274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:49.083446 8274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:49.083523 8274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:49.087801 8274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:49.087921 8274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:49.087996 8274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 09:05:49.088078 8274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:49.093701 8274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:49.094901 8274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:49.095997 8274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:49.105693 8282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:49.105693 8280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:49.105857 8279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:49.106206 8274 server_base.cc:1034] running on GCE node
I20260620 09:05:49.106660 8274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:49.107336 8274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:49.108494 8274 hybrid_clock.cc:648] HybridClock initialized: now 1781946349108481 us; error 37 us; skew 500 ppm
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946349, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:49.111732 8274 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:05:49.112686 8274 webserver.cc:460] Webserver started at http://127.0.98.3:38743/ using document root <none> and password file <none>
I20260620 09:05:49.113354 8274 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:49.113447 8274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:49.113695 8274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:49.115759 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ac3b08b8b2b4391912c8cb7d404c948"
format_stamp: "Formatted at 2026-06-20 09:05:49 on dist-test-slave-92t4"
server_key: "587c670ec31518aa388bae46f1918d96"
server_key_iv: "d9639f26d32c7d6dd4d6e24bb2f04987"
server_key_version: "encryptionkey@0"
I20260620 09:05:49.116412 8274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ac3b08b8b2b4391912c8cb7d404c948"
format_stamp: "Formatted at 2026-06-20 09:05:49 on dist-test-slave-92t4"
server_key: "587c670ec31518aa388bae46f1918d96"
server_key_iv: "d9639f26d32c7d6dd4d6e24bb2f04987"
server_key_version: "encryptionkey@0"
I20260620 09:05:49.120396 8274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:05:49.122974 8289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:49.124214 8274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:05:49.124361 8274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "2ac3b08b8b2b4391912c8cb7d404c948"
format_stamp: "Formatted at 2026-06-20 09:05:49 on dist-test-slave-92t4"
server_key: "587c670ec31518aa388bae46f1918d96"
server_key_iv: "d9639f26d32c7d6dd4d6e24bb2f04987"
server_key_version: "encryptionkey@0"
I20260620 09:05:49.124487 8274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:49.143115 8274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:49.146237 8274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:49.146456 8274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:49.147986 8274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:49.148059 8274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:49.148125 8274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:49.148180 8274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946349, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:49.158962 8274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38539
I20260620 09:05:49.159024 8404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38539 every 8 connection(s)
I20260620 09:05:49.159885 8274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:49.162942 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.148698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54881 (local address 127.0.98.62:35715)
0620 09:05:49.148820 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.148825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.149941 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 09:05:49.150999 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.151003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.151066 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.151130 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:49.152935 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.153698 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.155580 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.155957 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.156064 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.159310 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:49.159314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:49.159316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:49.159351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:49.161118 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.161600 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.161602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.161604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.161678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.161962 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.161963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.161964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.162135 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:49.162211 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.162721 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.162821 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:05:49.163820 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.149297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.3:54881)
0620 09:05:49.149779 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.149823 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.150728 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.151612 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.151622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.152106 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:49.152782 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.152801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.153833 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.153837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.155436 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.155451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.156212 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.156214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.156404 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.157115 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:49.157145 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:49.159123 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:49.161247 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.161252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.161263 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.161513 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.161758 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.161760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.161761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.161886 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.162249 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:49.162254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:49.162515 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.162679 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.162943 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260620 09:05:49.164963 8405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35715
I20260620 09:05:49.165122 8405 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:49.165670 8405 heartbeater.cc:511] Master 127.0.98.62:35715 requested a full tablet report, sending...
I20260620 09:05:49.166841 7921 ts_manager.cc:194] Registered new tserver with Master: 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539)
I20260620 09:05:49.166955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8274
I20260620 09:05:49.167062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:49.167420 392 external_mini_cluster.cc:1428] Setting key 72564d24e93f328012a1846cdbbba7bc
I20260620 09:05:49.167670 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54881
I20260620 09:05:49.169443 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:05:49.175061 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.167473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48619 (local address 127.0.98.62:35715)
0620 09:05:49.167585 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.167589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.167926 (+ 337us) server_negotiation.cc:407] Connection header received
0620 09:05:49.168079 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.168082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.168122 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.168219 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:49.168827 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.169592 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.170411 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.170644 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.170752 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.171844 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:49.171848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:49.171851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:49.171872 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:49.173444 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.173890 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.173894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.173896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.173979 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.174249 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.174262 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.174264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.174439 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:49.174546 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.174801 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.174906 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 09:05:49.175069 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.167405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.98.3:48619)
0620 09:05:49.167851 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.167866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.167965 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.168238 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.168241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.168600 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:49.168709 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.168717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.169720 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.169723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.170294 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.170301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.170753 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.170764 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.170852 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.171373 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:49.171384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:49.171693 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:49.173555 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.173560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.173562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.173789 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.174069 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.174073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.174075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.174161 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.174536 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:49.174538 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:49.174635 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.174738 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.174893 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:05:49.193946 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:49.187564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.0.98.62:35715)
0620 09:05:49.187695 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.187698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.187709 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:05:49.187746 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.187749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.187788 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.187877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:49.188578 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.189170 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.190042 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.190280 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.190389 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.193768 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60436: BlockingRecv error: recv got EOF from 127.0.0.1:60436 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:49.203068 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.195343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.98.62:35715)
0620 09:05:49.195470 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.195474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.195569 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:05:49.195707 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.195710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.195777 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.195859 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:49.196426 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.196938 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.197739 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.197957 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.198074 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.200307 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:49.200310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:49.200312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:49.200334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:49.201755 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.202116 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.202119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.202120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.202191 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.202415 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.202417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.202418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.202589 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:49.202682 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.202865 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.202953 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:05:49.205909 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:49.208997 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:49.223546 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.218134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.0.1:36198)
0620 09:05:49.218529 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.218540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.218611 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.219321 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.219327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.219361 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.219619 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.219630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.221094 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.221100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.222155 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.222163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.223005 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.223009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.223106 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.223142 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.223267 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.223394 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":61,"spinlock_wait_cycles":2432,"thread_start_us":81,"threads_started":1}
I20260620 09:05:49.223553 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.218565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.0.98.3:38539)
0620 09:05:49.218881 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.218891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.218917 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:05:49.218978 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.218982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.219154 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.219609 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.219748 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.220933 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.222290 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.222882 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.223008 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.223185 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.223312 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.223402 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260620 09:05:49.223855 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.217535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36115 (local address 127.0.0.1:44172)
0620 09:05:49.218439 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.218454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.218555 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.219320 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.219327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.219369 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.219557 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.219564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.220727 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.220733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.222225 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.222236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.223355 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.223359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.223456 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.223484 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.223537 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.223633 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":92,"spinlock_wait_cycles":10752,"thread_start_us":117,"threads_started":1}
I20260620 09:05:49.224359 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.217438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40529 (local address 127.0.0.1:37414)
0620 09:05:49.218215 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.218271 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.218424 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.219489 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.219493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.219510 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.219613 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.219620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.221076 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.221080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.222265 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.222276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.223545 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.223548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.224079 (+ 531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.224101 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.224166 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.224228 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":426,"thread_start_us":77,"threads_started":1}
I20260620 09:05:49.224587 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.217917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.0.98.2:36115)
0620 09:05:49.218237 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.218249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.218851 (+ 602us) server_negotiation.cc:407] Connection header received
0620 09:05:49.218938 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.218946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.219153 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.219359 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.219713 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.220621 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.222411 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.223216 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.224106 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.224269 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.224317 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.224425 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260620 09:05:49.225093 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.217581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.0.98.1:40529)
0620 09:05:49.218808 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.218824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.218848 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:05:49.218933 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.218940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.219186 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.219343 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.219747 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.220933 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.222414 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.223216 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.223334 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.223486 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.224727 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.224949 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":67,"threads_started":1}
I20260620 09:05:49.226972 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59f028d174e8381ebafd1a08500b9 (DEFAULT_TABLE table=test-table [id=7f083295435b4a4b9f652110c7d22b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.226966 8339 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59f028d174e8381ebafd1a08500b9 (DEFAULT_TABLE table=test-table [id=7f083295435b4a4b9f652110c7d22b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.227514 8059 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59f028d174e8381ebafd1a08500b9 (DEFAULT_TABLE table=test-table [id=7f083295435b4a4b9f652110c7d22b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.228070 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59f028d174e8381ebafd1a08500b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.228086 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59f028d174e8381ebafd1a08500b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.228453 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59f028d174e8381ebafd1a08500b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.234920 8433 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Bootstrap starting.
I20260620 09:05:49.236323 8434 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Bootstrap starting.
I20260620 09:05:49.237224 8433 tablet_bootstrap.cc:654] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.237393 8435 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Bootstrap starting.
I20260620 09:05:49.238080 8433 log.cc:824] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:49.238368 8434 tablet_bootstrap.cc:654] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.239463 8435 tablet_bootstrap.cc:654] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.239643 8434 log.cc:824] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:49.240212 8435 log.cc:824] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:49.240413 8433 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: No bootstrap required, opened a new log
I20260620 09:05:49.240629 8433 ts_tablet_manager.cc:1397] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 09:05:49.241624 8434 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: No bootstrap required, opened a new log
I20260620 09:05:49.241818 8434 ts_tablet_manager.cc:1397] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:49.242218 8435 tablet_bootstrap.cc:492] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: No bootstrap required, opened a new log
I20260620 09:05:49.242395 8435 ts_tablet_manager.cc:1397] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:05:49.243903 8433 raft_consensus.cc:348] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.244163 8433 raft_consensus.cc:374] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.244220 8433 raft_consensus.cc:729] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.244674 8433 consensus_queue.cc:260] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.245644 8265 heartbeater.cc:503] Master 127.0.98.62:35715 was elected leader, sending a full tablet report...
I20260620 09:05:49.245729 8433 ts_tablet_manager.cc:1428] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:05:49.246212 8435 raft_consensus.cc:348] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.246480 8435 raft_consensus.cc:374] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.246548 8435 raft_consensus.cc:729] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.246846 8434 raft_consensus.cc:348] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.247028 8435 consensus_queue.cc:260] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.247124 8434 raft_consensus.cc:374] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.247174 8434 raft_consensus.cc:729] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.247583 8434 consensus_queue.cc:260] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.247692 8405 heartbeater.cc:503] Master 127.0.98.62:35715 was elected leader, sending a full tablet report...
I20260620 09:05:49.248034 8435 ts_tablet_manager.cc:1428] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:05:49.248509 8434 ts_tablet_manager.cc:1428] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:05:49.248811 8125 heartbeater.cc:503] Master 127.0.98.62:35715 was elected leader, sending a full tablet report...
I20260620 09:05:49.363613 8441 raft_consensus.cc:482] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:49.363822 8441 raft_consensus.cc:504] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.364987 8441 leader_election.cc:288] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115), 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539)
I20260620 09:05:49.369108 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.365233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36115 (local address 127.0.98.1:58313)
0620 09:05:49.365618 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.365632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.365789 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.366118 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.366121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.366139 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.366261 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.366274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.367016 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.367022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.367987 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.367997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.368639 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.368743 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.368765 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.368862 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.368934 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":175,"threads_started":1}
I20260620 09:05:49.369093 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.365421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58313 (local address 127.0.98.2:36115)
0620 09:05:49.365605 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.365612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.365753 (+ 141us) server_negotiation.cc:407] Connection header received
0620 09:05:49.365934 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.365937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.365992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.366106 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.366382 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.366905 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368118 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.368520 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368638 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.368702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.368878 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.368952 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 09:05:49.369376 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.365899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52145 (local address 127.0.98.3:38539)
0620 09:05:49.366030 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.366034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.366210 (+ 176us) server_negotiation.cc:407] Connection header received
0620 09:05:49.366332 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.366335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.366380 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.366465 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.366736 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.367248 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368357 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.368759 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368871 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.368931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.369149 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.369240 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:05:49.369536 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.365766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.98.1:52145)
0620 09:05:49.366124 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.366136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.366247 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.366476 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.366479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.366493 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.366622 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.366627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.367390 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.367395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.368202 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.368211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.368875 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.368878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.368991 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.369012 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.369125 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.369331 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260620 09:05:49.369730 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59f028d174e8381ebafd1a08500b9" candidate_uuid: "94a345ba5aa74973a8b24af3fbf2b427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d" is_pre_election: true
I20260620 09:05:49.370092 8219 raft_consensus.cc:2454] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a345ba5aa74973a8b24af3fbf2b427 in term 0.
I20260620 09:05:49.370085 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59f028d174e8381ebafd1a08500b9" candidate_uuid: "94a345ba5aa74973a8b24af3fbf2b427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" is_pre_election: true
I20260620 09:05:49.370371 8359 raft_consensus.cc:2454] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a345ba5aa74973a8b24af3fbf2b427 in term 0.
I20260620 09:05:49.370601 8012 leader_election.cc:302] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:49.370914 8441 raft_consensus.cc:2781] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:49.370990 8441 raft_consensus.cc:482] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:49.371019 8441 raft_consensus.cc:3037] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.372051 8441 raft_consensus.cc:504] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.372392 8441 leader_election.cc:288] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [CANDIDATE]: Term 1 election: Requested vote from peers 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115), 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539)
I20260620 09:05:49.372786 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59f028d174e8381ebafd1a08500b9" candidate_uuid: "94a345ba5aa74973a8b24af3fbf2b427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac3b08b8b2b4391912c8cb7d404c948"
I20260620 09:05:49.372781 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59f028d174e8381ebafd1a08500b9" candidate_uuid: "94a345ba5aa74973a8b24af3fbf2b427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d"
I20260620 09:05:49.372907 8219 raft_consensus.cc:3037] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.372907 8359 raft_consensus.cc:3037] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.374241 8219 raft_consensus.cc:2454] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a345ba5aa74973a8b24af3fbf2b427 in term 1.
I20260620 09:05:49.374241 8359 raft_consensus.cc:2454] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a345ba5aa74973a8b24af3fbf2b427 in term 1.
I20260620 09:05:49.374527 8012 leader_election.cc:302] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:49.374688 8441 raft_consensus.cc:2781] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:49.374929 8441 raft_consensus.cc:686] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 LEADER]: Becoming Leader. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Running, Role: LEADER
I20260620 09:05:49.375272 8441 consensus_queue.cc:237] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.378379 7921 catalog_manager.cc:5495] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "94a345ba5aa74973a8b24af3fbf2b427" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } health_report { overall_health: UNKNOWN } } }
W20260620 09:05:49.411877 8406 tablet.cc:2369] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:49.418244 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.414631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.0.98.1:40529)
0620 09:05:49.414781 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.414786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.414815 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:05:49.414918 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.414921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.414977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.415088 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:49.415320 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.415834 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.416726 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.416979 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.417131 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.417398 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.417846 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.417977 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.418036 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.418094 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:05:49.424211 8359 raft_consensus.cc:1261] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Refusing update from remote peer 94a345ba5aa74973a8b24af3fbf2b427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.424211 8219 raft_consensus.cc:1261] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Refusing update from remote peer 94a345ba5aa74973a8b24af3fbf2b427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.424958 8441 consensus_queue.cc:1035] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.425266 8444 consensus_queue.cc:1035] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.433063 8448 mvcc.cc:205] Tried to move back new op lower bound from 7298852247234285568 to 7298852247044042752. Current Snapshot: MvccSnapshot[applied={T|T < 7298852247234285568}]
I20260620 09:05:49.441229 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:49.441423 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:49.443903 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Sending DeleteTablet for 3 replicas of tablet 77d59f028d174e8381ebafd1a08500b9
I20260620 09:05:49.444643 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d59f028d174e8381ebafd1a08500b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260620 09:05:49.444907 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d59f028d174e8381ebafd1a08500b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44172
I20260620 09:05:49.444947 8339 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d59f028d174e8381ebafd1a08500b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36198
I20260620 09:05:49.445235 8458 tablet_replica.cc:330] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: stopping tablet replica
I20260620 09:05:49.445585 8459 tablet_replica.cc:330] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: stopping tablet replica
I20260620 09:05:49.445587 8458 raft_consensus.cc:2229] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:49.445901 8459 raft_consensus.cc:2229] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:49.445935 8458 raft_consensus.cc:2258] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.446255 8459 raft_consensus.cc:2258] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.446228 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:49.446658 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:49.447801 8459 ts_tablet_manager.cc:1905] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.448499 8458 ts_tablet_manager.cc:1905] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.450845 8460 tablet_replica.cc:330] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: stopping tablet replica
I20260620 09:05:49.451201 8460 raft_consensus.cc:2229] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:49.451237 8458 ts_tablet_manager.cc:1918] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:49.451431 8458 log.cc:1196] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/77d59f028d174e8381ebafd1a08500b9
I20260620 09:05:49.451545 8460 raft_consensus.cc:2258] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.451880 8458 ts_tablet_manager.cc:1939] T 77d59f028d174e8381ebafd1a08500b9 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting consensus metadata
I20260620 09:05:49.451992 8459 ts_tablet_manager.cc:1918] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:49.452096 8459 log.cc:1196] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/77d59f028d174e8381ebafd1a08500b9
I20260620 09:05:49.452409 8459 ts_tablet_manager.cc:1939] T 77d59f028d174e8381ebafd1a08500b9 P 265f21c300f64f638d5a4508ede55e6d: Deleting consensus metadata
I20260620 09:05:49.452823 8059 tablet_service.cc:1467] Processing CreateTablet for tablet f2691070ab254153bbf20ab3f4b2379d (DEFAULT_TABLE table=test-table [id=44a8506dc9794f678b1fa7ab8423146b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.453102 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2691070ab254153bbf20ab3f4b2379d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.454669 8460 ts_tablet_manager.cc:1905] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.455305 8434 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Bootstrap starting.
I20260620 09:05:49.456214 8434 tablet_bootstrap.cc:654] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.457379 8434 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: No bootstrap required, opened a new log
I20260620 09:05:49.457397 8460 ts_tablet_manager.cc:1918] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:49.457480 8434 ts_tablet_manager.cc:1397] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:49.457480 7908 catalog_manager.cc:4841] TS 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115): tablet 77d59f028d174e8381ebafd1a08500b9 (table test-table [id=7f083295435b4a4b9f652110c7d22b11]) successfully deleted
I20260620 09:05:49.457633 8460 log.cc:1196] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/77d59f028d174e8381ebafd1a08500b9
I20260620 09:05:49.457993 8460 ts_tablet_manager.cc:1939] T 77d59f028d174e8381ebafd1a08500b9 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting consensus metadata
I20260620 09:05:49.457947 8434 raft_consensus.cc:348] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.458063 8434 raft_consensus.cc:374] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.458106 8434 raft_consensus.cc:729] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.458171 8199 tablet_service.cc:1467] Processing CreateTablet for tablet f2691070ab254153bbf20ab3f4b2379d (DEFAULT_TABLE table=test-table [id=44a8506dc9794f678b1fa7ab8423146b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.458232 8434 consensus_queue.cc:260] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.458424 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2691070ab254153bbf20ab3f4b2379d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.458509 8434 ts_tablet_manager.cc:1428] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:49.458689 7909 catalog_manager.cc:4841] TS 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529): tablet 77d59f028d174e8381ebafd1a08500b9 (table test-table [id=7f083295435b4a4b9f652110c7d22b11]) successfully deleted
I20260620 09:05:49.459482 7908 catalog_manager.cc:4841] TS 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539): tablet 77d59f028d174e8381ebafd1a08500b9 (table test-table [id=7f083295435b4a4b9f652110c7d22b11]) successfully deleted
I20260620 09:05:49.460778 8433 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Bootstrap starting.
I20260620 09:05:49.461119 8339 tablet_service.cc:1467] Processing CreateTablet for tablet f2691070ab254153bbf20ab3f4b2379d (DEFAULT_TABLE table=test-table [id=44a8506dc9794f678b1fa7ab8423146b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:49.461407 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2691070ab254153bbf20ab3f4b2379d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.461719 8433 tablet_bootstrap.cc:654] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.463239 8433 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: No bootstrap required, opened a new log
I20260620 09:05:49.463317 8433 ts_tablet_manager.cc:1397] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:05:49.463382 8435 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Bootstrap starting.
I20260620 09:05:49.463723 8433 raft_consensus.cc:348] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.463828 8433 raft_consensus.cc:374] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.463858 8433 raft_consensus.cc:729] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.464178 8433 consensus_queue.cc:260] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.464355 8435 tablet_bootstrap.cc:654] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.464920 8433 ts_tablet_manager.cc:1428] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:49.471859 8435 tablet_bootstrap.cc:492] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: No bootstrap required, opened a new log
I20260620 09:05:49.471957 8435 ts_tablet_manager.cc:1397] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 09:05:49.472405 8435 raft_consensus.cc:348] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.472541 8435 raft_consensus.cc:374] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.472573 8435 raft_consensus.cc:729] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Initialized, Role: FOLLOWER
W20260620 09:05:49.472646 8126 tablet.cc:2369] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:05:49.472692 8435 consensus_queue.cc:260] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.472942 8435 ts_tablet_manager.cc:1428] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 09:05:49.643545 8440 raft_consensus.cc:482] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:49.643716 8440 raft_consensus.cc:504] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.644950 8440 leader_election.cc:288] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:49.650082 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.645280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33811 (local address 127.0.98.1:40529)
0620 09:05:49.645413 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.645418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.645531 (+ 113us) server_negotiation.cc:407] Connection header received
0620 09:05:49.645663 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.645669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.645728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.645817 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.646165 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.646944 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.648733 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.649367 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.649476 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.649543 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.649874 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.649950 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:05:49.650082 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.645348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50269 (local address 127.0.98.2:36115)
0620 09:05:49.645477 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.645481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.645648 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:05:49.645733 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.645735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.645788 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.645858 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:49.646165 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.646944 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.648733 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.649367 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.649476 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.649541 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.649874 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.649948 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:05:49.650113 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.645199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40529 (local address 127.0.98.3:33811)
0620 09:05:49.645416 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.645432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.645536 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.645864 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.645869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.645886 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.646048 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.646055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.647066 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.647069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.648611 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.648623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.649497 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.649504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.649722 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.649768 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.649846 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.649937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":301,"spinlock_wait_cycles":7424}
I20260620 09:05:49.650125 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.645285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36115 (local address 127.0.98.3:50269)
0620 09:05:49.645546 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.645558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.645649 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.645881 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.645883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.645894 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:49.646049 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.646055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.647066 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.647069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.648620 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.648625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.649512 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.649515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.649694 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.649763 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.649849 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.649937 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":248,"thread_start_us":97,"threads_started":1}
I20260620 09:05:49.650688 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2691070ab254153bbf20ab3f4b2379d" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427" is_pre_election: true
I20260620 09:05:49.650822 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2691070ab254153bbf20ab3f4b2379d" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d" is_pre_election: true
I20260620 09:05:49.650945 8219 raft_consensus.cc:2454] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:49.650966 8079 raft_consensus.cc:2454] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:49.651306 8292 leader_election.cc:302] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 2ac3b08b8b2b4391912c8cb7d404c948; no voters:
I20260620 09:05:49.651585 8440 raft_consensus.cc:2781] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:49.651647 8440 raft_consensus.cc:482] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:49.651685 8440 raft_consensus.cc:3037] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.652664 8440 raft_consensus.cc:504] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.652990 8440 leader_election.cc:288] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Requested vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:49.653424 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2691070ab254153bbf20ab3f4b2379d" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d"
I20260620 09:05:49.653492 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2691070ab254153bbf20ab3f4b2379d" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427"
I20260620 09:05:49.653550 8219 raft_consensus.cc:3037] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.653625 8079 raft_consensus.cc:3037] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.654611 8219 raft_consensus.cc:2454] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:49.654690 8079 raft_consensus.cc:2454] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:49.654907 8292 leader_election.cc:302] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 2ac3b08b8b2b4391912c8cb7d404c948; no voters:
I20260620 09:05:49.655071 8440 raft_consensus.cc:2781] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:49.655248 8440 raft_consensus.cc:686] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Running, Role: LEADER
I20260620 09:05:49.655488 8440 consensus_queue.cc:237] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.657593 7922 catalog_manager.cc:5495] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } health_report { overall_health: HEALTHY } } }
W20260620 09:05:49.691668 8266 tablet.cc:2369] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 09:05:49 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946348, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:49.739316 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.730774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35715 (local address 127.0.0.1:60448)
0620 09:05:49.731049 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.731065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.731218 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.731474 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.731477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.731879 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:49.731973 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.731979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.732748 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.732751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.733361 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.733369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.733830 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.733833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.733903 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.734492 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:49.734504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:49.736034 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:49.737861 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.737868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.737873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.738098 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.738364 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:49.738366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:49.738368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:49.738452 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:49.738792 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:49.738797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:49.738907 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.738998 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.739137 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260620 09:05:49.739324 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.730836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.98.62:35715)
0620 09:05:49.731056 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.731062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.731157 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:05:49.731321 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.731324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.731363 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.731461 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:49.732089 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.732625 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.733479 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.733721 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.733828 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.736201 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:49.736204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:49.736206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:49.736227 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:49.737761 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.738183 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.738186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.738187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.738256 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:49.738536 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:49.738538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:49.738539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:49.738703 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:49.738777 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.739051 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.739193 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260620 09:05:49.743284 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 09:05:49.744066 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:49.748935 8059 tablet_service.cc:1467] Processing CreateTablet for tablet c427001e2dbe4ea690bac2178611c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61378d91d7e340d4abcf516481b0e1e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:49.748934 8199 tablet_service.cc:1467] Processing CreateTablet for tablet c427001e2dbe4ea690bac2178611c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61378d91d7e340d4abcf516481b0e1e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:49.748970 8339 tablet_service.cc:1467] Processing CreateTablet for tablet c427001e2dbe4ea690bac2178611c1b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61378d91d7e340d4abcf516481b0e1e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 09:05:49.749354 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c427001e2dbe4ea690bac2178611c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.749354 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c427001e2dbe4ea690bac2178611c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.749409 8339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c427001e2dbe4ea690bac2178611c1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:49.751928 8435 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: Bootstrap starting.
I20260620 09:05:49.752120 8434 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: Bootstrap starting.
I20260620 09:05:49.752144 8433 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: Bootstrap starting.
I20260620 09:05:49.752763 8435 tablet_bootstrap.cc:654] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.752959 8434 tablet_bootstrap.cc:654] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.753000 8433 tablet_bootstrap.cc:654] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:49.754304 8434 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: No bootstrap required, opened a new log
I20260620 09:05:49.754329 8435 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: No bootstrap required, opened a new log
I20260620 09:05:49.754400 8434 ts_tablet_manager.cc:1397] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:49.754431 8435 ts_tablet_manager.cc:1397] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 09:05:49.754588 8433 tablet_bootstrap.cc:492] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: No bootstrap required, opened a new log
I20260620 09:05:49.754667 8433 ts_tablet_manager.cc:1397] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:05:49.754817 8434 raft_consensus.cc:348] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.754874 8435 raft_consensus.cc:348] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.754961 8434 raft_consensus.cc:374] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.754983 8435 raft_consensus.cc:374] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.754998 8434 raft_consensus.cc:729] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.755025 8435 raft_consensus.cc:729] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.755100 8433 raft_consensus.cc:348] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.755121 8434 consensus_queue.cc:260] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.755205 8433 raft_consensus.cc:374] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:49.755151 8435 consensus_queue.cc:260] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.755246 8433 raft_consensus.cc:729] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Initialized, Role: FOLLOWER
I20260620 09:05:49.755393 8434 ts_tablet_manager.cc:1428] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:49.755352 8433 consensus_queue.cc:260] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.755424 8435 ts_tablet_manager.cc:1428] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 09:05:49.755606 8433 ts_tablet_manager.cc:1428] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:49.755609 8444 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.755615 8440 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.755748 8440 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.755745 8444 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.755947 8439 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.756076 8439 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.871685 8440 raft_consensus.cc:482] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:49.871845 8440 raft_consensus.cc:504] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.872296 8440 leader_election.cc:288] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:49.872787 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c427001e2dbe4ea690bac2178611c1b7" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427" is_pre_election: true
I20260620 09:05:49.872856 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c427001e2dbe4ea690bac2178611c1b7" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d" is_pre_election: true
I20260620 09:05:49.872962 8079 raft_consensus.cc:2454] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:49.873008 8219 raft_consensus.cc:2454] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:49.873296 8293 leader_election.cc:302] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac3b08b8b2b4391912c8cb7d404c948, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:49.873468 8440 raft_consensus.cc:2781] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:49.873556 8440 raft_consensus.cc:482] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:49.873610 8440 raft_consensus.cc:3037] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.874631 8440 raft_consensus.cc:504] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.874981 8440 leader_election.cc:288] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Requested vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:49.875345 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c427001e2dbe4ea690bac2178611c1b7" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d"
I20260620 09:05:49.875351 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c427001e2dbe4ea690bac2178611c1b7" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427"
I20260620 09:05:49.875445 8219 raft_consensus.cc:3037] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.875483 8079 raft_consensus.cc:3037] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:49.876351 8219 raft_consensus.cc:2454] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:49.876399 8079 raft_consensus.cc:2454] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:49.876650 8292 leader_election.cc:302] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 2ac3b08b8b2b4391912c8cb7d404c948; no voters:
I20260620 09:05:49.876821 8440 raft_consensus.cc:2781] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:49.876935 8440 raft_consensus.cc:686] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Running, Role: LEADER
I20260620 09:05:49.877096 8440 consensus_queue.cc:237] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:49.877784 8465 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: TxnStatusTablet state changed. Reason: New leader 2ac3b08b8b2b4391912c8cb7d404c948. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.877916 8465 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:49.878154 8478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 09:05:49.878352 8478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 09:05:49.878964 7920 catalog_manager.cc:5495] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:49.913429 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.909479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.0.98.3:38539)
0620 09:05:49.909807 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.909820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.909836 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:05:49.910008 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.910010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.910074 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.910188 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:49.910453 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.910998 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.911849 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.912099 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.912212 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.912504 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.912984 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.913127 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.913217 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.913269 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260620 09:05:49.913437 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.909368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.0.1:36210)
0620 09:05:49.909702 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.909719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.909830 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.910206 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.910209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.910218 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:49.910322 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.910328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.911108 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.911113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.911729 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.911737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.912210 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.912214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.912304 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.912355 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.913104 (+ 749us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.913121 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.913207 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.913298 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 09:05:49.916519 8219 raft_consensus.cc:1261] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.916785 8079 raft_consensus.cc:1261] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.917425 8465 consensus_queue.cc:1035] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.917613 8440 consensus_queue.cc:1035] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.919117 8439 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: TxnStatusTablet state changed. Reason: New leader 2ac3b08b8b2b4391912c8cb7d404c948. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.919230 8439 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.919500 8444 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: TxnStatusTablet state changed. Reason: New leader 2ac3b08b8b2b4391912c8cb7d404c948. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.919621 8444 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.921152 8440 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.921283 8440 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:49.922428 8447 mvcc.cc:205] Tried to move back new op lower bound from 7298852249253122048 to 7298852249098199040. Current Snapshot: MvccSnapshot[applied={T|T < 7298852249253122048}]
I20260620 09:05:49.922587 8444 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.922672 8444 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 94a345ba5aa74973a8b24af3fbf2b427: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.924757 8465 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.924872 8465 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 2ac3b08b8b2b4391912c8cb7d404c948: This TxnStatusTablet replica's current role is: LEADER
I20260620 09:05:49.925609 8439 tablet_replica.cc:439] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } } }
I20260620 09:05:49.925716 8439 tablet_replica.cc:442] T c427001e2dbe4ea690bac2178611c1b7 P 265f21c300f64f638d5a4508ede55e6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 09:05:49.930825 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.927299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.0.98.3:38539)
0620 09:05:49.927421 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.927425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.927650 (+ 225us) server_negotiation.cc:407] Connection header received
0620 09:05:49.927818 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.927821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.927866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.927950 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:49.928202 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.928833 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.929751 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.929989 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.930111 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.930356 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.930442 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.930522 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.930619 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.930672 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:05:49.941958 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.938016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50331 (local address 127.0.98.3:38539)
0620 09:05:49.938122 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:49.938125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:49.938395 (+ 270us) server_negotiation.cc:407] Connection header received
0620 09:05:49.938498 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:49.938501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:49.938549 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:49.938628 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:49.938891 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.939558 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.940793 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.941145 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.941240 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.941530 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.941606 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.941673 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:49.941783 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 09:05:49.941830 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:05:49.942016 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:49.937947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.98.3:50331)
0620 09:05:49.938293 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:49.938307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:49.938408 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:49.938650 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:49.938653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:49.938660 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:49.938783 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.938789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.939666 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.939670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.940672 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:49.940682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:49.941254 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:49.941258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:49.941353 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:49.941402 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:49.941692 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:49.941697 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:05:49.941763 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:05:49.941846 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":95,"threads_started":1}
I20260620 09:05:49.949150 8219 raft_consensus.cc:1261] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.949226 8079 raft_consensus.cc:1261] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:49.949617 8465 consensus_queue.cc:1035] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.949858 8440 consensus_queue.cc:1035] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:49.958218 8449 mvcc.cc:205] Tried to move back new op lower bound from 7298852249388523520 to 7298852248191188992. Current Snapshot: MvccSnapshot[applied={T|T < 7298852249388523520}]
I20260620 09:05:49.990765 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:49.990911 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:49.993117 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Sending DeleteTablet for 3 replicas of tablet f2691070ab254153bbf20ab3f4b2379d
I20260620 09:05:49.993669 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet f2691070ab254153bbf20ab3f4b2379d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260620 09:05:49.993997 8494 tablet_replica.cc:330] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: stopping tablet replica
I20260620 09:05:49.993973 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet f2691070ab254153bbf20ab3f4b2379d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44172
I20260620 09:05:49.994014 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet f2691070ab254153bbf20ab3f4b2379d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:49 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36198
I20260620 09:05:49.994258 8494 raft_consensus.cc:2229] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:49.994395 8494 raft_consensus.cc:2258] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.994573 8496 tablet_replica.cc:330] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: stopping tablet replica
I20260620 09:05:49.994690 8496 raft_consensus.cc:2229] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:49.994971 8496 raft_consensus.cc:2258] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.995107 8494 ts_tablet_manager.cc:1905] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.995805 8496 ts_tablet_manager.cc:1905] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.997203 8495 tablet_replica.cc:330] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: stopping tablet replica
I20260620 09:05:49.997331 8495 raft_consensus.cc:2229] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:49.997454 8495 raft_consensus.cc:2258] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:49.998564 8494 ts_tablet_manager.cc:1918] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:49.998652 8494 log.cc:1196] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/f2691070ab254153bbf20ab3f4b2379d
I20260620 09:05:49.998658 8495 ts_tablet_manager.cc:1905] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:49.998986 8494 ts_tablet_manager.cc:1939] T f2691070ab254153bbf20ab3f4b2379d P 94a345ba5aa74973a8b24af3fbf2b427: Deleting consensus metadata
I20260620 09:05:49.999680 7909 catalog_manager.cc:4841] TS 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529): tablet f2691070ab254153bbf20ab3f4b2379d (table test-table [id=44a8506dc9794f678b1fa7ab8423146b]) successfully deleted
I20260620 09:05:50.001088 8495 ts_tablet_manager.cc:1918] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:50.001179 8495 log.cc:1196] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/f2691070ab254153bbf20ab3f4b2379d
I20260620 09:05:50.001472 8495 ts_tablet_manager.cc:1939] T f2691070ab254153bbf20ab3f4b2379d P 265f21c300f64f638d5a4508ede55e6d: Deleting consensus metadata
I20260620 09:05:50.002230 7908 catalog_manager.cc:4841] TS 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115): tablet f2691070ab254153bbf20ab3f4b2379d (table test-table [id=44a8506dc9794f678b1fa7ab8423146b]) successfully deleted
I20260620 09:05:50.002779 8496 ts_tablet_manager.cc:1918] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:50.002944 8496 log.cc:1196] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/f2691070ab254153bbf20ab3f4b2379d
I20260620 09:05:50.003295 8496 ts_tablet_manager.cc:1939] T f2691070ab254153bbf20ab3f4b2379d P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting consensus metadata
I20260620 09:05:50.004009 7908 catalog_manager.cc:4841] TS 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539): tablet f2691070ab254153bbf20ab3f4b2379d (table test-table [id=44a8506dc9794f678b1fa7ab8423146b]) successfully deleted
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[7877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:05:50.025089 8473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:50.018063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.98.62:35715)
0620 09:05:50.018225 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.018230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.018489 (+ 259us) server_negotiation.cc:407] Connection header received
0620 09:05:50.018600 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.018604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.018671 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.018773 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:50.019434 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.020293 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.021213 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.021558 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.021676 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.024899 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60462: BlockingRecv error: recv got EOF from 127.0.0.1:60462 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[7877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 09:05:50.035686 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.026970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.0.98.62:35715)
0620 09:05:50.027090 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.027101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.027183 (+ 82us) server_negotiation.cc:407] Connection header received
0620 09:05:50.027323 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.027325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.027364 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.027432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:50.028182 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.028770 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.029746 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.029966 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.030073 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.032611 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:50.032614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:50.032616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:50.032641 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:50.034061 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:50.034472 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:50.034477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:50.034478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:50.034560 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:50.034825 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:50.034829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:50.034831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:50.035017 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:50.035161 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.035382 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.035512 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:50.038218 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:05:50.038668 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:50.043485 8059 tablet_service.cc:1467] Processing CreateTablet for tablet bd1e4e9809584a0f9f820f807f334d22 (DEFAULT_TABLE table=test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.043592 8338 tablet_service.cc:1467] Processing CreateTablet for tablet bd1e4e9809584a0f9f820f807f334d22 (DEFAULT_TABLE table=test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.043828 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1e4e9809584a0f9f820f807f334d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.043874 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1e4e9809584a0f9f820f807f334d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.046077 8434 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Bootstrap starting.
I20260620 09:05:50.046308 8199 tablet_service.cc:1467] Processing CreateTablet for tablet bd1e4e9809584a0f9f820f807f334d22 (DEFAULT_TABLE table=test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.046496 8435 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Bootstrap starting.
I20260620 09:05:50.046572 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1e4e9809584a0f9f820f807f334d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.046890 8434 tablet_bootstrap.cc:654] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.047286 8435 tablet_bootstrap.cc:654] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.048647 8434 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: No bootstrap required, opened a new log
I20260620 09:05:50.048748 8434 ts_tablet_manager.cc:1397] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 09:05:50.048993 8433 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Bootstrap starting.
I20260620 09:05:50.049202 8434 raft_consensus.cc:348] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.049325 8434 raft_consensus.cc:374] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.049360 8434 raft_consensus.cc:729] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.049485 8434 consensus_queue.cc:260] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.049769 8434 ts_tablet_manager.cc:1428] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.049849 8433 tablet_bootstrap.cc:654] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.050575 8435 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: No bootstrap required, opened a new log
I20260620 09:05:50.050662 8435 ts_tablet_manager.cc:1397] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 09:05:50.051051 8435 raft_consensus.cc:348] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.051183 8435 raft_consensus.cc:374] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.051223 8435 raft_consensus.cc:729] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.051342 8435 consensus_queue.cc:260] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.051620 8435 ts_tablet_manager.cc:1428] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.052476 8433 tablet_bootstrap.cc:492] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: No bootstrap required, opened a new log
I20260620 09:05:50.052552 8433 ts_tablet_manager.cc:1397] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 09:05:50.053195 8433 raft_consensus.cc:348] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.053293 8433 raft_consensus.cc:374] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.053321 8433 raft_consensus.cc:729] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.053432 8433 consensus_queue.cc:260] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.053669 8433 ts_tablet_manager.cc:1428] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.139921 8484 raft_consensus.cc:482] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:50.140074 8484 raft_consensus.cc:504] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.140535 8484 leader_election.cc:288] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:50.141016 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1e4e9809584a0f9f820f807f334d22" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427" is_pre_election: true
I20260620 09:05:50.141081 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1e4e9809584a0f9f820f807f334d22" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d" is_pre_election: true
I20260620 09:05:50.141237 8219 raft_consensus.cc:2454] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:50.141242 8079 raft_consensus.cc:2454] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 0.
I20260620 09:05:50.141608 8293 leader_election.cc:302] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac3b08b8b2b4391912c8cb7d404c948, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:50.141958 8484 raft_consensus.cc:2781] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:50.142017 8484 raft_consensus.cc:482] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:50.142063 8484 raft_consensus.cc:3037] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.143097 8484 raft_consensus.cc:504] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.143494 8484 leader_election.cc:288] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Requested vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115)
I20260620 09:05:50.143795 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1e4e9809584a0f9f820f807f334d22" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427"
I20260620 09:05:50.143860 8219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1e4e9809584a0f9f820f807f334d22" candidate_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265f21c300f64f638d5a4508ede55e6d"
I20260620 09:05:50.143941 8079 raft_consensus.cc:3037] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.143980 8219 raft_consensus.cc:3037] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.144986 8079 raft_consensus.cc:2454] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:50.145135 8219 raft_consensus.cc:2454] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac3b08b8b2b4391912c8cb7d404c948 in term 1.
I20260620 09:05:50.145334 8293 leader_election.cc:302] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac3b08b8b2b4391912c8cb7d404c948, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:50.145516 8484 raft_consensus.cc:2781] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:50.145599 8484 raft_consensus.cc:686] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Running, Role: LEADER
I20260620 09:05:50.145745 8484 consensus_queue.cc:237] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.147485 7922 catalog_manager.cc:5495] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } health_report { overall_health: HEALTHY } } }
I20260620 09:05:50.173168 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.169988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.98.3:38539)
0620 09:05:50.170107 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.170111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.170174 (+ 63us) server_negotiation.cc:407] Connection header received
0620 09:05:50.170330 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.170333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.170388 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.170469 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:50.170715 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.171233 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.172097 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.172321 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.172426 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.172689 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.172767 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.172858 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.172941 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.172994 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 09:05:50.175158 8079 raft_consensus.cc:1261] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:50.175141 8219 raft_consensus.cc:1261] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Refusing update from remote peer 2ac3b08b8b2b4391912c8cb7d404c948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:50.175689 8484 consensus_queue.cc:1035] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:50.175850 8465 consensus_queue.cc:1035] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:50.184335 7922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:50.184520 7922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:50.186872 7922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Sending DeleteTablet for 3 replicas of tablet bd1e4e9809584a0f9f820f807f334d22
I20260620 09:05:50.187714 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1e4e9809584a0f9f820f807f334d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260620 09:05:50.187703 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1e4e9809584a0f9f820f807f334d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44172
I20260620 09:05:50.187879 8494 tablet_replica.cc:330] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: stopping tablet replica
I20260620 09:05:50.187984 8494 raft_consensus.cc:2229] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:50.188083 8494 raft_consensus.cc:2258] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.188243 8495 tablet_replica.cc:330] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: stopping tablet replica
I20260620 09:05:50.188354 8495 raft_consensus.cc:2229] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:50.188452 8495 raft_consensus.cc:2258] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.189023 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1e4e9809584a0f9f820f807f334d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36198
I20260620 09:05:50.189224 8496 tablet_replica.cc:330] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: stopping tablet replica
I20260620 09:05:50.189345 8496 raft_consensus.cc:2229] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:50.189347 8494 ts_tablet_manager.cc:1905] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.189505 8496 raft_consensus.cc:2258] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.190310 8496 ts_tablet_manager.cc:1905] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.192046 8494 ts_tablet_manager.cc:1918] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:50.192121 8494 log.cc:1196] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/bd1e4e9809584a0f9f820f807f334d22
I20260620 09:05:50.192373 8494 ts_tablet_manager.cc:1939] T bd1e4e9809584a0f9f820f807f334d22 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting consensus metadata
I20260620 09:05:50.192883 8496 ts_tablet_manager.cc:1918] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:50.192947 8496 log.cc:1196] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/bd1e4e9809584a0f9f820f807f334d22
I20260620 09:05:50.193202 8496 ts_tablet_manager.cc:1939] T bd1e4e9809584a0f9f820f807f334d22 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting consensus metadata
I20260620 09:05:50.193686 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 09:05:50.193840 7908 catalog_manager.cc:4841] TS 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539): tablet bd1e4e9809584a0f9f820f807f334d22 (table test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]) successfully deleted
W20260620 09:05:50.194175 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:05:50.194813 8495 ts_tablet_manager.cc:1905] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.197147 8495 ts_tablet_manager.cc:1918] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:05:50.197227 8495 log.cc:1196] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/bd1e4e9809584a0f9f820f807f334d22
I20260620 09:05:50.197455 8495 ts_tablet_manager.cc:1939] T bd1e4e9809584a0f9f820f807f334d22 P 265f21c300f64f638d5a4508ede55e6d: Deleting consensus metadata
I20260620 09:05:50.198191 7908 catalog_manager.cc:4841] TS 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115): tablet bd1e4e9809584a0f9f820f807f334d22 (table test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]) successfully deleted
I20260620 09:05:50.198205 7909 catalog_manager.cc:4841] TS 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529): tablet bd1e4e9809584a0f9f820f807f334d22 (table test-table [id=087ffa7b7dc845a9ba60e810eddbc29f]) successfully deleted
I20260620 09:05:50.201756 8059 tablet_service.cc:1467] Processing CreateTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 (DEFAULT_TABLE table=test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.201855 8199 tablet_service.cc:1467] Processing CreateTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 (DEFAULT_TABLE table=test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.202050 8059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab319f68e3b4daca9a6e11a1dd5faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.202114 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab319f68e3b4daca9a6e11a1dd5faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.204170 8433 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Bootstrap starting.
I20260620 09:05:50.204519 8434 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Bootstrap starting.
I20260620 09:05:50.205150 8433 tablet_bootstrap.cc:654] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.205317 8434 tablet_bootstrap.cc:654] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.206049 8338 tablet_service.cc:1467] Processing CreateTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 (DEFAULT_TABLE table=test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:05:50.206338 8433 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: No bootstrap required, opened a new log
I20260620 09:05:50.206338 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab319f68e3b4daca9a6e11a1dd5faa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:50.206421 8433 ts_tablet_manager.cc:1397] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:50.206893 8433 raft_consensus.cc:348] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.207010 8433 raft_consensus.cc:374] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.207039 8433 raft_consensus.cc:729] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.207154 8433 consensus_queue.cc:260] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.207442 8433 ts_tablet_manager.cc:1428] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.207962 8434 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: No bootstrap required, opened a new log
I20260620 09:05:50.208050 8434 ts_tablet_manager.cc:1397] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 09:05:50.208472 8434 raft_consensus.cc:348] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.208603 8434 raft_consensus.cc:374] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.208637 8434 raft_consensus.cc:729] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a345ba5aa74973a8b24af3fbf2b427, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.208757 8434 consensus_queue.cc:260] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.209208 8435 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Bootstrap starting.
I20260620 09:05:50.209019 8434 ts_tablet_manager.cc:1428] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.210022 8435 tablet_bootstrap.cc:654] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:50.211158 8435 tablet_bootstrap.cc:492] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: No bootstrap required, opened a new log
I20260620 09:05:50.211256 8435 ts_tablet_manager.cc:1397] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:50.211658 8435 raft_consensus.cc:348] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.211786 8435 raft_consensus.cc:374] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:50.211841 8435 raft_consensus.cc:729] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac3b08b8b2b4391912c8cb7d404c948, State: Initialized, Role: FOLLOWER
I20260620 09:05:50.211973 8435 consensus_queue.cc:260] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.212244 8435 ts_tablet_manager.cc:1428] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:05:50.322863 8439 raft_consensus.cc:482] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:05:50.323014 8439 raft_consensus.cc:504] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.324239 8439 leader_election.cc:288] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539)
I20260620 09:05:50.328437 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.324609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60871 (local address 127.0.98.3:38539)
0620 09:05:50.324760 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.324764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.324904 (+ 140us) server_negotiation.cc:407] Connection header received
0620 09:05:50.325081 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.325084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.325133 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.325208 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:50.325566 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.326240 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327381 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.327869 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327984 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.328047 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.328241 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.328310 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 09:05:50.328451 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.324413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40529 (local address 127.0.98.2:44899)
0620 09:05:50.324755 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:50.324771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:50.324906 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:50.325262 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:50.325265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:50.325281 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:50.325444 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.325449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.326284 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.326287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.327253 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.327261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.327943 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.328048 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.328071 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:05:50.328169 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:05:50.328244 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":44,"thread_start_us":85,"threads_started":1}
I20260620 09:05:50.328452 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.324533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44899 (local address 127.0.98.1:40529)
0620 09:05:50.324773 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.324779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.324856 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:05:50.325077 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.325083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.325137 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.325265 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:05:50.325557 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.326157 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327387 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.327812 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327935 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.328003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.328183 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.328258 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
I20260620 09:05:50.328912 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab319f68e3b4daca9a6e11a1dd5faa5" candidate_uuid: "265f21c300f64f638d5a4508ede55e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427" is_pre_election: true
I20260620 09:05:50.329113 8079 raft_consensus.cc:2454] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265f21c300f64f638d5a4508ede55e6d in term 0.
I20260620 09:05:50.329386 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.324481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.98.2:60871)
0620 09:05:50.324798 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:50.324811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:50.324939 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:50.325229 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:50.325232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:50.325264 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:05:50.325461 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.325467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.326340 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.326346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.327258 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.327267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.327984 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.327988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.328112 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.328134 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:05:50.328226 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 09:05:50.328295 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":20,"spinlock_wait_cycles":21120,"thread_start_us":88,"threads_started":1}
I20260620 09:05:50.329543 8153 leader_election.cc:302] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 94a345ba5aa74973a8b24af3fbf2b427; no voters:
I20260620 09:05:50.329818 8439 raft_consensus.cc:2781] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:05:50.329880 8439 raft_consensus.cc:482] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:05:50.329928 8439 raft_consensus.cc:3037] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.329861 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab319f68e3b4daca9a6e11a1dd5faa5" candidate_uuid: "265f21c300f64f638d5a4508ede55e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" is_pre_election: true
I20260620 09:05:50.330014 8359 raft_consensus.cc:2454] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 265f21c300f64f638d5a4508ede55e6d in term 0.
I20260620 09:05:50.331000 8439 raft_consensus.cc:504] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.331367 8439 leader_election.cc:288] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [CANDIDATE]: Term 1 election: Requested vote from peers 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529), 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539)
I20260620 09:05:50.331682 8359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab319f68e3b4daca9a6e11a1dd5faa5" candidate_uuid: "265f21c300f64f638d5a4508ede55e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac3b08b8b2b4391912c8cb7d404c948"
I20260620 09:05:50.331799 8359 raft_consensus.cc:3037] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.332635 8359 raft_consensus.cc:2454] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265f21c300f64f638d5a4508ede55e6d in term 1.
I20260620 09:05:50.332867 8079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab319f68e3b4daca9a6e11a1dd5faa5" candidate_uuid: "265f21c300f64f638d5a4508ede55e6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a345ba5aa74973a8b24af3fbf2b427"
I20260620 09:05:50.332904 8152 leader_election.cc:302] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265f21c300f64f638d5a4508ede55e6d, 2ac3b08b8b2b4391912c8cb7d404c948; no voters:
I20260620 09:05:50.332988 8079 raft_consensus.cc:3037] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:50.333130 8439 raft_consensus.cc:2781] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:50.333343 8439 raft_consensus.cc:686] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 1 LEADER]: Becoming Leader. State: Replica: 265f21c300f64f638d5a4508ede55e6d, State: Running, Role: LEADER
I20260620 09:05:50.333591 8439 consensus_queue.cc:237] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } }
I20260620 09:05:50.334007 8079 raft_consensus.cc:2454] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 265f21c300f64f638d5a4508ede55e6d in term 1.
I20260620 09:05:50.336089 7921 catalog_manager.cc:5495] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 265f21c300f64f638d5a4508ede55e6d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "265f21c300f64f638d5a4508ede55e6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265f21c300f64f638d5a4508ede55e6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 } health_report { overall_health: UNKNOWN } } }
I20260620 09:05:50.366678 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.363116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.0.98.2:36115)
0620 09:05:50.363426 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.363433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.363445 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:05:50.363521 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.363523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.363575 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.363683 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:50.363849 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.364357 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.365231 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.365483 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.365590 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.365768 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.366271 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.366366 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.366424 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.366470 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":158,"threads_started":1}
I20260620 09:05:50.376665 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.373266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38539 (local address 127.0.98.2:57509)
0620 09:05:50.373569 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:50.373586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:50.373688 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:50.373951 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:50.373954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:50.373961 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:50.374077 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.374082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.374798 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.374804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.375446 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.375454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.375933 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.375939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.376023 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.376065 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.376355 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.376363 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:05:50.376431 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 09:05:50.376510 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":108,"threads_started":1}
I20260620 09:05:50.376636 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.373490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57509 (local address 127.0.98.3:38539)
0620 09:05:50.373640 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.373645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.373661 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:05:50.373805 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.373808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.373862 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.373941 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:50.374197 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.374691 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.375568 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.375790 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.375894 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.376202 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.376273 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.376352 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.376439 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.376486 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:05:50.382700 8359 raft_consensus.cc:1261] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Refusing update from remote peer 265f21c300f64f638d5a4508ede55e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:50.382704 8079 raft_consensus.cc:1261] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Refusing update from remote peer 265f21c300f64f638d5a4508ede55e6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:05:50.383379 8522 consensus_queue.cc:1035] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac3b08b8b2b4391912c8cb7d404c948" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:50.383564 8439 consensus_queue.cc:1035] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a345ba5aa74973a8b24af3fbf2b427" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:05:50.400745 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.397526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47611 (local address 127.0.98.2:36115)
0620 09:05:50.397656 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:50.397660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:50.397676 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:05:50.397824 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:50.397827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:50.397887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:50.397971 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:05:50.398142 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.398662 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.399574 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.399933 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.400026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.400307 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.400390 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.400469 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:50.400544 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 09:05:50.400599 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:50.400775 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:50.397486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36115 (local address 127.0.98.3:47611)
0620 09:05:50.397614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:50.397629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:50.397712 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:50.397975 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:50.397977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:50.397983 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 09:05:50.398070 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.398075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.398756 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.398758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.399362 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:50.399370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:50.400049 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:50.400053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:50.400134 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:50.400173 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 09:05:50.400480 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 09:05:50.400486 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:05:50.400533 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:05:50.400605 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 09:05:50.419589 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:50.419762 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:05:50.421906 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69aa97e99b043628b8f38d859d1b1aa: Sending DeleteTablet for 3 replicas of tablet bab319f68e3b4daca9a6e11a1dd5faa5
I20260620 09:05:50.422469 8059 tablet_service.cc:1514] Processing DeleteTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37414
I20260620 09:05:50.422611 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44172
I20260620 09:05:50.422713 8494 tablet_replica.cc:330] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: stopping tablet replica
I20260620 09:05:50.422796 8495 tablet_replica.cc:330] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: stopping tablet replica
I20260620 09:05:50.422847 8494 raft_consensus.cc:2229] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:50.422922 8495 raft_consensus.cc:2229] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:05:50.422973 8494 raft_consensus.cc:2258] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.423229 8495 raft_consensus.cc:2258] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.423894 8494 ts_tablet_manager.cc:1905] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.424041 8495 ts_tablet_manager.cc:1905] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.426473 8494 ts_tablet_manager.cc:1918] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:50.426564 8494 log.cc:1196] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/bab319f68e3b4daca9a6e11a1dd5faa5
I20260620 09:05:50.426600 8495 ts_tablet_manager.cc:1918] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:50.426678 8495 log.cc:1196] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/bab319f68e3b4daca9a6e11a1dd5faa5
I20260620 09:05:50.426831 8494 ts_tablet_manager.cc:1939] T bab319f68e3b4daca9a6e11a1dd5faa5 P 94a345ba5aa74973a8b24af3fbf2b427: Deleting consensus metadata
I20260620 09:05:50.426954 8495 ts_tablet_manager.cc:1939] T bab319f68e3b4daca9a6e11a1dd5faa5 P 265f21c300f64f638d5a4508ede55e6d: Deleting consensus metadata
I20260620 09:05:50.427582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7994
I20260620 09:05:50.427567 7909 catalog_manager.cc:4841] TS 94a345ba5aa74973a8b24af3fbf2b427 (127.0.98.1:40529): tablet bab319f68e3b4daca9a6e11a1dd5faa5 (table test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]) successfully deleted
I20260620 09:05:50.427626 7908 catalog_manager.cc:4841] TS 265f21c300f64f638d5a4508ede55e6d (127.0.98.2:36115): tablet bab319f68e3b4daca9a6e11a1dd5faa5 (table test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]) successfully deleted
I20260620 09:05:50.427847 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet bab319f68e3b4daca9a6e11a1dd5faa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:05:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36198
I20260620 09:05:50.428007 8496 tablet_replica.cc:330] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: stopping tablet replica
I20260620 09:05:50.428117 8496 raft_consensus.cc:2229] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:05:50.428231 8496 raft_consensus.cc:2258] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:05:50.429991 8496 ts_tablet_manager.cc:1905] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:05:50.432411 8496 ts_tablet_manager.cc:1918] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 09:05:50.432498 8496 log.cc:1196] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/bab319f68e3b4daca9a6e11a1dd5faa5
I20260620 09:05:50.432745 8496 ts_tablet_manager.cc:1939] T bab319f68e3b4daca9a6e11a1dd5faa5 P 2ac3b08b8b2b4391912c8cb7d404c948: Deleting consensus metadata
I20260620 09:05:50.433455 7908 catalog_manager.cc:4841] TS 2ac3b08b8b2b4391912c8cb7d404c948 (127.0.98.3:38539): tablet bab319f68e3b4daca9a6e11a1dd5faa5 (table test-table [id=4dec4ec45fe641bfa21671a0fce3dab7]) successfully deleted
I20260620 09:05:50.436007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8134
I20260620 09:05:50.442178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8274
I20260620 09:05:50.448832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 7890
2026-06-20T09:05:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3341 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[8534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[8534](info): set up 2 sockets
Jun 20 09:05:50 dist-test-slave-92t4 krb5kdc[8534](info): commencing operation
krb5kdc: starting...
W20260620 09:05:51.477944 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:51 dist-test-slave-92t4 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:51.602209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:51.710803 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:51.711127 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:51.711201 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:51.715169 8547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:51.715276 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:51.715322 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:51.715378 8547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:51.715415 8547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:51.720623 8547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:51.721794 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:51.722815 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:51.731396 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:51.731415 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:51.731415 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:51.731899 8547 server_base.cc:1034] running on GCE node
I20260620 09:05:51.732470 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:51.733673 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:51.734856 8547 hybrid_clock.cc:648] HybridClock initialized: now 1781946351734828 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T09:05:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1290 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:05:51 dist-test-slave-92t4 krb5kdc[8560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:05:51 dist-test-slave-92t4 krb5kdc[8560](info): set up 2 sockets
Jun 20 09:05:51 dist-test-slave-92t4 krb5kdc[8560](info): commencing operation
krb5kdc: starting...
W20260620 09:05:52.777892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:05:52 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:05:52.895627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:53.004813 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:53.005204 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:53.005257 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:53.009398 8573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:53.009472 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:53.009492 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:53.009509 8573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:53.009526 8573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:53.014976 8573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:53.016053 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:53.017089 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:53.025400 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.025406 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.025453 8573 server_base.cc:1034] running on GCE node
W20260620 09:05:53.025444 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.026196 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:53.027251 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:53.028427 8573 hybrid_clock.cc:648] HybridClock initialized: now 1781946353028414 us; error 27 us; skew 500 ppm
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:53.035876 8573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:53.036819 8573 webserver.cc:460] Webserver started at http://127.0.98.62:35933/ using document root <none> and password file <none>
I20260620 09:05:53.037453 8573 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:53.037518 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:53.037690 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:53.039739 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "d22e203fb3f9433fa37c0769e99ea3c9"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "e2927689ebae8c41269507b07592900c"
server_key_iv: "e2d3e8b292ce632dc42637e0f1b8d628"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.040256 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "d22e203fb3f9433fa37c0769e99ea3c9"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "e2927689ebae8c41269507b07592900c"
server_key_iv: "e2d3e8b292ce632dc42637e0f1b8d628"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.043892 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:05:53.046283 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.047554 8573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:05:53.047652 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "d22e203fb3f9433fa37c0769e99ea3c9"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "e2927689ebae8c41269507b07592900c"
server_key_iv: "e2d3e8b292ce632dc42637e0f1b8d628"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.047734 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:53.061774 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:53.065119 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:53.065282 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:53.072166 8573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34637
I20260620 09:05:53.072185 8640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34637 every 8 connection(s)
I20260620 09:05:53.073197 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:53.075937 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:05:53.081578 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Bootstrap starting.
I20260620 09:05:53.081919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8573
I20260620 09:05:53.082041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:05:53.082327 392 external_mini_cluster.cc:1428] Setting key c8b85ca3c184a66b0cbf2d9a5fb8ba26
I20260620 09:05:53.084372 8641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Neither blocks nor log segments found. Creating new log.
I20260620 09:05:53.085182 8641 log.cc:824] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:53.087018 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: No bootstrap required, opened a new log
I20260620 09:05:53.090459 8641 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:53.090718 8641 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:05:53.090797 8641 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22e203fb3f9433fa37c0769e99ea3c9, State: Initialized, Role: FOLLOWER
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:53.091305 8641 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:53.091481 8641 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:53.091557 8641 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:53.091660 8641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:05:53.092551 8641 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:53.092900 8641 leader_election.cc:302] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22e203fb3f9433fa37c0769e99ea3c9; no voters:
I20260620 09:05:53.093226 8641 leader_election.cc:288] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:05:53.093338 8646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:05:53.093525 8646 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 LEADER]: Becoming Leader. State: Replica: d22e203fb3f9433fa37c0769e99ea3c9, State: Running, Role: LEADER
I20260620 09:05:53.093941 8646 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:53.094601 8641 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:53.095619 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22e203fb3f9433fa37c0769e99ea3c9. Latest consensus state: current_term: 1 leader_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260620 09:05:53.095810 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:53.096195 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260620 09:05:53.096519 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:53.097926 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.083742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.0.98.62:34637)
0620 09:05:53.084182 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:53.084197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:53.084233 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:05:53.084953 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:53.084973 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:53.085401 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:53.085740 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:53.086519 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.087573 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.088433 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.088911 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.089120 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.091780 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:53.091786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:53.091803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:53.091838 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:53.093928 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.094524 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.094530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.094536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.094669 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.094983 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.094989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.094991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.095540 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:53.095777 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:53.096570 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 09:05:53.096900 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
W20260620 09:05:53.100320 8661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:05:53.100405 8661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:05:53.100474 8655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:53.101262 8655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:53.106866 8655 catalog_manager.cc:1347] Generated new cluster ID: 9302ca64691949f48147bf14eaa8c59b
I20260620 09:05:53.106932 8655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:53.128111 8655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:05:53.129174 8655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:53.138345 8655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Generated new TSK 0
I20260620 09:05:53.139096 8655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:05:53.176445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:53.286656 8667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:53.286991 8667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:53.287070 8667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:53.291159 8667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:53.291270 8667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:53.291370 8667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:53.296854 8667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:53.297947 8667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:53.298952 8667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:53.308503 8675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.308523 8672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.308562 8667 server_base.cc:1034] running on GCE node
W20260620 09:05:53.308512 8673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.309214 8667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:53.309851 8667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:53.311002 8667 hybrid_clock.cc:648] HybridClock initialized: now 1781946353310986 us; error 26 us; skew 500 ppm
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:53.313781 8667 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:05:53.314590 8667 webserver.cc:460] Webserver started at http://127.0.98.1:46193/ using document root <none> and password file <none>
I20260620 09:05:53.315088 8667 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:53.315142 8667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:53.315304 8667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:53.317340 8667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc8a25f460584fc89d7a523e0d7d740c"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "eaa1f9fe86bba3594a92962fec48aa14"
server_key_iv: "b121f3374c0a05b2c996501bc5ef7485"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.317840 8667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc8a25f460584fc89d7a523e0d7d740c"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "eaa1f9fe86bba3594a92962fec48aa14"
server_key_iv: "b121f3374c0a05b2c996501bc5ef7485"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.321347 8667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:53.323575 8682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.324723 8667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:53.324856 8667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "bc8a25f460584fc89d7a523e0d7d740c"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "eaa1f9fe86bba3594a92962fec48aa14"
server_key_iv: "b121f3374c0a05b2c996501bc5ef7485"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.324960 8667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:53.351853 8667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:53.355304 8667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:53.355504 8667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:53.356112 8667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:53.356949 8667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:53.357020 8667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.357100 8667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:53.357154 8667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.365793 8667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32867
I20260620 09:05:53.365818 8795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32867 every 8 connection(s)
I20260620 09:05:53.366677 8667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:53.373237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8667
I20260620 09:05:53.373368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:05:53.373651 392 external_mini_cluster.cc:1428] Setting key c08bd3d4ac91897360b8bc05c662803e
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:53.380498 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.368519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51979 (local address 127.0.98.62:34637)
0620 09:05:53.368682 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:53.368686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:53.369523 (+ 837us) server_negotiation.cc:407] Connection header received
0620 09:05:53.370874 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:53.370876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:53.370920 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:53.370995 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:53.372117 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.372638 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.373920 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.374235 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.374350 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.376817 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:53.376823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:53.376830 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:53.376868 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:53.378378 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.378960 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.378963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.378964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.379032 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.379423 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.379426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.379427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.379595 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:53.379666 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:53.380288 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 09:05:53.380382 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 09:05:53.381759 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.368915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:51979)
0620 09:05:53.369370 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:53.369411 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:53.370670 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:53.371153 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:53.371160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:53.371491 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:53.371978 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.371990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.372733 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.372735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.373802 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.373810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.374370 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.374397 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.374584 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.375277 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:53.375292 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:53.376656 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:53.378491 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.378499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.378516 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.378867 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.379121 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.379127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.379132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.379327 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.379697 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:53.379705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:53.380027 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 09:05:53.380260 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 09:05:53.380626 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260620 09:05:53.383018 8796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260620 09:05:53.383261 8796 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:53.383736 8796 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:05:53.385262 8605 ts_manager.cc:194] Registered new tserver with Master: bc8a25f460584fc89d7a523e0d7d740c (127.0.98.1:32867)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:05:53.386477 8605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:05:53.399673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:53.508131 8801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:53.508391 8801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:53.508432 8801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:53.512427 8801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:53.512492 8801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:53.512547 8801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:53.517623 8801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:53.518586 8801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:53.519596 8801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:53.528657 8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.528662 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.528667 8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.528946 8801 server_base.cc:1034] running on GCE node
I20260620 09:05:53.529452 8801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:53.530071 8801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:53.531219 8801 hybrid_clock.cc:648] HybridClock initialized: now 1781946353531201 us; error 30 us; skew 500 ppm
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:53.533829 8801 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:05:53.534590 8801 webserver.cc:460] Webserver started at http://127.0.98.2:46583/ using document root <none> and password file <none>
I20260620 09:05:53.535053 8801 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:53.535105 8801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:53.535256 8801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:53.537209 8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "f052e69fdfd24ff58d5d76e6f951c408"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "4ea0033177ae2ad839eb35eefa5a510e"
server_key_iv: "b321a83d8bd816ce8be0efdb10b23a60"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.537696 8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f052e69fdfd24ff58d5d76e6f951c408"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "4ea0033177ae2ad839eb35eefa5a510e"
server_key_iv: "b321a83d8bd816ce8be0efdb10b23a60"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.541136 8801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:53.543359 8816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.544548 8801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:53.544672 8801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "f052e69fdfd24ff58d5d76e6f951c408"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "4ea0033177ae2ad839eb35eefa5a510e"
server_key_iv: "b321a83d8bd816ce8be0efdb10b23a60"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.544772 8801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:53.571331 8801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:53.574396 8801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:53.574654 8801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:53.575356 8801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:53.576400 8801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:53.576458 8801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.576519 8801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:53.576552 8801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.585191 8801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33795
I20260620 09:05:53.585227 8929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33795 every 8 connection(s)
I20260620 09:05:53.586084 8801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:53.595747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8801
I20260620 09:05:53.595878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:05:53.596175 392 external_mini_cluster.cc:1428] Setting key 648a291b5d8400f213c11fc4d0707b24
I20260620 09:05:53.599494 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.587797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57235 (local address 127.0.98.62:34637)
0620 09:05:53.587928 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:53.587931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:53.588735 (+ 804us) server_negotiation.cc:407] Connection header received
0620 09:05:53.590124 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:53.590127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:53.590171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:53.590244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:53.591318 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.591823 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.592878 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.593141 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.593226 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.595685 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:53.595689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:53.595692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:53.595724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:53.597486 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.597972 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.597975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.597978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.598071 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.598377 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.598384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.598389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.598633 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:53.598734 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:53.599174 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 09:05:53.599319 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:05:53.600533 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.588168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:57235)
0620 09:05:53.588589 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:53.588630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:53.589939 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:53.590397 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:53.590403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:53.590732 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:53.591202 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.591214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.591922 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.591925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.592780 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.592787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.593269 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.593272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.593410 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.593984 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:53.593999 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:53.595527 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:53.597598 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.597604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.597618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.597871 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.598164 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.598166 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.598167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.598288 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.598728 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:53.598732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:53.598950 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 09:05:53.599098 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 09:05:53.599459 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
I20260620 09:05:53.601894 8930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260620 09:05:53.602146 8930 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:53.602712 8930 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
I20260620 09:05:53.603808 8605 ts_manager.cc:194] Registered new tserver with Master: f052e69fdfd24ff58d5d76e6f951c408 (127.0.98.2:33795)
I20260620 09:05:53.604375 8605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57235
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:05:53.623561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:53.736560 8935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:53.737190 8935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:53.737285 8935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:53.745589 8935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:53.745661 8935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:53.745764 8935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:53.753643 8935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:53.754803 8935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:53.756026 8935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:53.768383 8941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.768422 8943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:53.768402 8940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:53.769021 8935 server_base.cc:1034] running on GCE node
I20260620 09:05:53.769546 8935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:53.770190 8935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:53.771368 8935 hybrid_clock.cc:648] HybridClock initialized: now 1781946353771338 us; error 48 us; skew 500 ppm
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:53.774643 8935 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:05:53.775614 8935 webserver.cc:460] Webserver started at http://127.0.98.3:39863/ using document root <none> and password file <none>
I20260620 09:05:53.776158 8935 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:53.776247 8935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:53.776480 8935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:05:53.778537 8935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4e5f20c4f3144628d558181e2aa0a31"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "b5fd66e3240fd04120d386b4f916d462"
server_key_iv: "8c40895bf8af47fa91105c30f699dbbe"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.779086 8935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4e5f20c4f3144628d558181e2aa0a31"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "b5fd66e3240fd04120d386b4f916d462"
server_key_iv: "8c40895bf8af47fa91105c30f699dbbe"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.782645 8935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 09:05:53.784960 8950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.786140 8935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:53.786296 8935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "e4e5f20c4f3144628d558181e2aa0a31"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "b5fd66e3240fd04120d386b4f916d462"
server_key_iv: "8c40895bf8af47fa91105c30f699dbbe"
server_key_version: "encryptionkey@0"
I20260620 09:05:53.786401 8935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:53.808099 8935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:53.811421 8935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:53.811614 8935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:53.812170 8935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:53.812953 8935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:53.813023 8935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.813110 8935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:53.813159 8935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:53.822497 8935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44405
I20260620 09:05:53.822523 9063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44405 every 8 connection(s)
I20260620 09:05:53.823436 8935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:53.831351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 8935
I20260620 09:05:53.831487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:05:53.831873 392 external_mini_cluster.cc:1428] Setting key 9fd74cc90e25fa6b0af9ac9ed33cfe48
Jun 20 09:05:53 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:05:53.836822 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.825151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41105 (local address 127.0.98.62:34637)
0620 09:05:53.825281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:53.825286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:53.826094 (+ 808us) server_negotiation.cc:407] Connection header received
0620 09:05:53.827468 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:53.827471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:53.827523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:53.827628 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:53.828848 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.829491 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.830631 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.830854 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.830960 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.833778 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:53.833781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:53.833783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:53.833805 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:53.835205 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.835628 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.835631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.835632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.835700 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:53.835971 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:53.835973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:53.835974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:53.836127 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:53.836198 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:53.836615 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 09:05:53.836703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:53.837744 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:53.825531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:41105)
0620 09:05:53.825942 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:53.825981 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:53.827263 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:53.827810 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:53.827817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:53.828161 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:53.828699 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.828713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.829623 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.829632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.830508 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:53.830517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:53.830996 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:53.831000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:53.831253 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:53.831874 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:53.831896 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:53.833621 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:05:53.835301 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.835305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.835316 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.835546 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.835780 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:05:53.835782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:05:53.835783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:05:53.835900 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:05:53.836208 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:05:53.836212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:05:53.836429 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 09:05:53.836573 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 09:05:53.836861 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260620 09:05:53.838805 9064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34637
I20260620 09:05:53.839020 9064 heartbeater.cc:464] Registering TS with master...
I20260620 09:05:53.839430 9064 heartbeater.cc:511] Master 127.0.98.62:34637 requested a full tablet report, sending...
I20260620 09:05:53.840334 8605 ts_manager.cc:194] Registered new tserver with Master: e4e5f20c4f3144628d558181e2aa0a31 (127.0.98.3:44405)
I20260620 09:05:53.840866 8605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41105
I20260620 09:05:53.845734 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 09:05:53.912891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8667
I20260620 09:05:53.918251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8801
I20260620 09:05:53.923398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8935
I20260620 09:05:53.928725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 8573
I20260620 09:05:53.934723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--webserver_interface=127.0.98.62
--webserver_port=35933
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:05:54.044219 9073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:54.044517 9073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:54.044598 9073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:54.048588 9073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:05:54.048668 9073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:54.048705 9073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:54.048758 9073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:05:54.048810 9073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:05:54.054023 9073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:54.055056 9073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:54.056058 9073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:54.064503 9079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.064559 9081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.064579 9078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.064635 9073 server_base.cc:1034] running on GCE node
I20260620 09:05:54.065356 9073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:54.066422 9073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:54.067605 9073 hybrid_clock.cc:648] HybridClock initialized: now 1781946354067589 us; error 28 us; skew 500 ppm
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:54.070374 9073 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:05:54.071250 9073 webserver.cc:460] Webserver started at http://127.0.98.62:35933/ using document root <none> and password file <none>
I20260620 09:05:54.071812 9073 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:54.071923 9073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:54.075703 9073 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:54.077875 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.079007 9073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:05:54.079144 9073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "d22e203fb3f9433fa37c0769e99ea3c9"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "e2927689ebae8c41269507b07592900c"
server_key_iv: "e2d3e8b292ce632dc42637e0f1b8d628"
server_key_version: "encryptionkey@0"
I20260620 09:05:54.079668 9073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:54.104697 9073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:54.107594 9073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:54.107810 9073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:54.114948 9073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34637
I20260620 09:05:54.114961 9140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34637 every 8 connection(s)
I20260620 09:05:54.115979 9073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:05:54.120159 9141 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:05:54.121057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9073
I20260620 09:05:54.121557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32867
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34637
--webserver_port=46193
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:05:54.122151 9141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Bootstrap starting.
I20260620 09:05:54.141390 9141 log.cc:824] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:05:54.147989 9141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:05:54.148360 9141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Bootstrap complete.
I20260620 09:05:54.152580 9141 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:54.152978 9141 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22e203fb3f9433fa37c0769e99ea3c9, State: Initialized, Role: FOLLOWER
I20260620 09:05:54.153473 9141 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:54.153585 9141 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:05:54.153666 9141 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:05:54.153754 9141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:05:54.155704 9141 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:54.156039 9141 leader_election.cc:302] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22e203fb3f9433fa37c0769e99ea3c9; no voters:
I20260620 09:05:54.156366 9141 leader_election.cc:288] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:05:54.156461 9145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:05:54.156649 9145 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [term 2 LEADER]: Becoming Leader. State: Replica: d22e203fb3f9433fa37c0769e99ea3c9, State: Running, Role: LEADER
I20260620 09:05:54.156935 9145 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } }
I20260620 09:05:54.157612 9141 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:05:54.158281 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260620 09:05:54.158403 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:54.159035 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22e203fb3f9433fa37c0769e99ea3c9. Latest consensus state: current_term: 2 leader_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22e203fb3f9433fa37c0769e99ea3c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } }
I20260620 09:05:54.159195 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:05:54.159557 9152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:05:54.162659 9152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:05:54.163260 9152 catalog_manager.cc:1259] Loaded cluster ID: 9302ca64691949f48147bf14eaa8c59b
I20260620 09:05:54.163322 9152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:05:54.165665 9152 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:05:54.166289 9152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d22e203fb3f9433fa37c0769e99ea3c9: Loaded TSK: 0
I20260620 09:05:54.167501 9152 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:05:54.238023 9143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:54.238314 9143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:54.238389 9143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:54.242455 9143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:54.242535 9143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:54.242624 9143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:05:54.247967 9143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46193
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:54.248972 9143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:54.250000 9143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:54.260366 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.260406 9167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.260406 9168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.261176 9143 server_base.cc:1034] running on GCE node
I20260620 09:05:54.261610 9143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:54.262358 9143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:54.263562 9143 hybrid_clock.cc:648] HybridClock initialized: now 1781946354263526 us; error 45 us; skew 500 ppm
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946354, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:54.267050 9143 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 09:05:54.268157 9143 webserver.cc:460] Webserver started at http://127.0.98.1:46193/ using document root <none> and password file <none>
I20260620 09:05:54.268860 9143 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:54.268971 9143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:54.273953 9143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:05:54.276399 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.277902 9143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:05:54.278028 9143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "bc8a25f460584fc89d7a523e0d7d740c"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "eaa1f9fe86bba3594a92962fec48aa14"
server_key_iv: "b121f3374c0a05b2c996501bc5ef7485"
server_key_version: "encryptionkey@0"
I20260620 09:05:54.278710 9143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:54.305172 9143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:54.308679 9143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:54.308876 9143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:54.309559 9143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:54.310586 9143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:54.310652 9143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.310707 9143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:54.310736 9143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.320358 9143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32867
I20260620 09:05:54.320396 9290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32867 every 8 connection(s)
I20260620 09:05:54.321411 9143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:05:54.328341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9143
I20260620 09:05:54.328842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33795
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34637
--webserver_port=46583
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.334626 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.323530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:57893)
0620 09:05:54.323974 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.324027 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.325440 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.326483 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.326489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.326823 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.327292 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.327306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.328768 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.328771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.329754 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.329763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.330596 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.330601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.330842 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.331554 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.331570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.333888 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
W20260620 09:05:54.335107 9291 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34637 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34637: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260620 09:05:54.335142 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.323371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57893 (local address 127.0.98.62:34637)
0620 09:05:54.323803 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.323819 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.324145 (+ 326us) server_negotiation.cc:407] Connection header received
0620 09:05:54.325798 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.325809 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.326099 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.326390 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.327455 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.328648 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.329885 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.330429 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.330546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.334194 (+ 3648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57893: BlockingRecv error: recv got EOF from 127.0.98.1:57893 (error 108)
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.341282 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.335562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:47563)
0620 09:05:54.335792 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.335803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.335880 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.336128 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.336130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.336257 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.336350 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.336355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.337473 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.337476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.338016 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.338023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.338679 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.338681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.338747 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.339285 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.339297 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.341128 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":85}
W20260620 09:05:54.341391 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.335613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47563 (local address 127.0.98.62:34637)
0620 09:05:54.335760 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.335764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.335866 (+ 102us) server_negotiation.cc:407] Connection header received
0620 09:05:54.335950 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.335954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.336013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.336113 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.336461 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.337370 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.338131 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.338560 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.338681 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.341258 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47563: BlockingRecv error: recv got EOF from 127.0.98.1:47563 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.346695 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.341914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:39985)
0620 09:05:54.342043 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.342066 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.342129 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.342361 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.342363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.342481 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.342548 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.342552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.343272 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.343278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.343791 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.343798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.344227 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.344231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.344306 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.344785 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.344795 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.346568 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 09:05:54.346891 9291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:05:54.347030 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.341998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39985 (local address 127.0.98.62:34637)
0620 09:05:54.342106 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.342110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.342124 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:54.342207 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.342209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.342253 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.342332 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.342671 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.343167 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.343904 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.344121 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.344226 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.346685 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39985: BlockingRecv error: recv got EOF from 127.0.98.1:39985 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 09:05:54.441767 9295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:54.442027 9295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:54.442070 9295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:54.446241 9295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:54.446297 9295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:54.446354 9295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:05:54.451658 9295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46583
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:54.452689 9295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:54.453740 9295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:54.463799 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.463873 9295 server_base.cc:1034] running on GCE node
W20260620 09:05:54.463817 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.463795 9303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.464478 9295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:54.465081 9295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:54.466251 9295 hybrid_clock.cc:648] HybridClock initialized: now 1781946354466238 us; error 26 us; skew 500 ppm
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946354, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:54.468837 9295 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 09:05:54.469734 9295 webserver.cc:460] Webserver started at http://127.0.98.2:46583/ using document root <none> and password file <none>
I20260620 09:05:54.470320 9295 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:54.470427 9295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:54.474213 9295 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 09:05:54.476178 9310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.477433 9295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 09:05:54.477557 9295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "f052e69fdfd24ff58d5d76e6f951c408"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "4ea0033177ae2ad839eb35eefa5a510e"
server_key_iv: "b321a83d8bd816ce8be0efdb10b23a60"
server_key_version: "encryptionkey@0"
I20260620 09:05:54.478080 9295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:54.509349 9295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:54.512319 9295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:54.512532 9295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:54.513198 9295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:54.514006 9295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:54.514074 9295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.514145 9295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:54.514186 9295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.522855 9295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33795
I20260620 09:05:54.522887 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33795 every 8 connection(s)
I20260620 09:05:54.523849 9295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:05:54.525736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9295
I20260620 09:05:54.526178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44405
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34637
--webserver_port=39863
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:05:54.536885 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.525656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36651 (local address 127.0.98.62:34637)
0620 09:05:54.525785 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.525789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.526820 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 09:05:54.528652 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.528655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.528707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.528786 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.530463 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.530967 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.532811 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.533019 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.533160 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.536768 (+ 3608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36651: BlockingRecv error: recv got EOF from 127.0.98.2:36651 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:05:54.537598 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.526180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:36651)
0620 09:05:54.526663 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.526704 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.528393 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.529029 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.529066 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.529575 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.530298 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.530320 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.531091 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.531098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.532679 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.532694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.533217 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.533223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.533457 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.534068 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.534111 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.536651 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
W20260620 09:05:54.538057 9424 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34637 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34637: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.543577 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.538533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:41811)
0620 09:05:54.538653 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.538668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.538741 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.538969 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.538971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.539090 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.539170 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.539174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.540150 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.540152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.540720 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.540728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.541266 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.541269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.541334 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.541807 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.541817 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.543444 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 09:05:54.543887 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.538618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41811 (local address 127.0.98.62:34637)
0620 09:05:54.538729 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.538733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.538747 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:05:54.538826 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.538836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.538877 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.538948 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.539265 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.540052 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.540834 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.541175 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.541271 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.543571 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41811: BlockingRecv error: recv got EOF from 127.0.98.2:41811 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.548919 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.544075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:47329)
0620 09:05:54.544194 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.544208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.544282 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.544512 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.544514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.544637 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.544699 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.544703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.545638 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.545640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.546149 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.546155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.546679 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.546681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.546734 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.547194 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.547205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.548796 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:05:54.549121 9424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:05:54.548991 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.544207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47329 (local address 127.0.98.62:34637)
0620 09:05:54.544326 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.544330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.544343 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:54.544377 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.544379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.544417 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.544493 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.544791 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.545550 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.546250 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.546581 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.546679 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.548894 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47329: BlockingRecv error: recv got EOF from 127.0.98.2:47329 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 09:05:54.641889 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:05:54.642186 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:05:54.642275 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:05:54.646298 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:05:54.646402 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:05:54.646499 9427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:05:54.651789 9427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39863
--tserver_master_addrs=127.0.98.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:05:54.652917 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:05:54.653975 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:05:54.663681 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.663790 9427 server_base.cc:1034] running on GCE node
W20260620 09:05:54.663681 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:05:54.663683 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:05:54.664371 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:05:54.664963 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:05:54.666114 9427 hybrid_clock.cc:648] HybridClock initialized: now 1781946354666102 us; error 31 us; skew 500 ppm
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946354, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:05:54.668924 9427 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 09:05:54.669952 9427 webserver.cc:460] Webserver started at http://127.0.98.3:39863/ using document root <none> and password file <none>
I20260620 09:05:54.670516 9427 fs_manager.cc:359] Metadata directory not provided
I20260620 09:05:54.670588 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:05:54.674515 9427 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 09:05:54.676551 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.677845 9427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:05:54.677976 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "e4e5f20c4f3144628d558181e2aa0a31"
format_stamp: "Formatted at 2026-06-20 09:05:53 on dist-test-slave-92t4"
server_key: "b5fd66e3240fd04120d386b4f916d462"
server_key_iv: "8c40895bf8af47fa91105c30f699dbbe"
server_key_version: "encryptionkey@0"
I20260620 09:05:54.678540 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:05:54.692950 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:05:54.696312 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:05:54.696527 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:05:54.697170 9427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:05:54.697981 9427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:05:54.698051 9427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.698141 9427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:05:54.698184 9427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:05:54.707060 9427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44405
I20260620 09:05:54.707113 9555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44405 every 8 connection(s)
I20260620 09:05:54.708339 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:05:54.713076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9427
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 09:05:54.720098 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.710121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36047 (local address 127.0.98.62:34637)
0620 09:05:54.710328 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.710332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.711145 (+ 813us) server_negotiation.cc:407] Connection header received
0620 09:05:54.712470 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.712472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.712520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.712607 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.714155 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.714748 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.716110 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.716330 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.716406 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.719998 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36047: BlockingRecv error: recv got EOF from 127.0.98.3:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 09:05:54.720893 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.710558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:36047)
0620 09:05:54.710991 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.711032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.712244 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.712834 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.712845 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.713390 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.713957 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.713976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.714874 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.714878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.715996 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.716010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.716474 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.716478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.716662 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.717323 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.717344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.719846 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
W20260620 09:05:54.721387 9556 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34637 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34637: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 09:05:54.723919 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.714138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.0.98.62:34637)
0620 09:05:54.714388 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.714391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.714400 (+ 9us) server_negotiation.cc:407] Connection header received
0620 09:05:54.714428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.714429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.714463 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.714560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.715425 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.715910 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.717008 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.717379 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.717457 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.718921 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:05:54.718928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:05:54.718946 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:05:54.718975 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:05:54.720941 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:54.721439 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:54.721447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:54.721454 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:54.721589 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:05:54.722154 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:05:54.722157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:05:54.722158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:05:54.722501 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:05:54.722641 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:05:54.723557 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 09:05:54.723787 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.727094 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.721879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:47367)
0620 09:05:54.721992 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.722007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.722076 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.722268 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.722271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.722423 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.722505 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.722509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.723204 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.723210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.723923 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.723932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.724510 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.724512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.724578 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.725012 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.725023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.727010 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:05:54.727263 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.721979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47367 (local address 127.0.98.62:34637)
0620 09:05:54.722088 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.722091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.722108 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:05:54.722134 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.722136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.722166 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.722251 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.722616 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.723102 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.724090 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.724405 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.724504 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.727128 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47367: BlockingRecv error: recv got EOF from 127.0.98.3:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:54 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:54.732085 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:54.727708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:47261)
0620 09:05:54.727812 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:54.727824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:54.727912 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:54.728151 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:54.728154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:54.728301 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:54.728366 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.728371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.729086 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.729092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.729612 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:54.729620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.730046 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.730049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:54.730123 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.730510 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:54.730520 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:54.732001 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 09:05:54.732210 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:54.727799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47261 (local address 127.0.98.62:34637)
0620 09:05:54.727968 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:54.727973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:54.727986 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:54.728024 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:54.728026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:54.728057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:54.728137 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:54.728472 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.728951 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.729729 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:54.729939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:54.730048 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:54.732109 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47261: BlockingRecv error: recv got EOF from 127.0.98.3:47261 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 09:05:54.732388 9556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:55.353243 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:55.347603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:47825)
0620 09:05:55.347885 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:55.347909 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:55.347996 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:55.348231 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:55.348233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:55.348411 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:55.348520 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.348525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.349321 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.349327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.350144 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.350153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.350743 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.350745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.350823 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.351304 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:55.351314 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:55.353123 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":67,"threads_started":1}
W20260620 09:05:55.353397 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:55.347683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47825 (local address 127.0.98.62:34637)
0620 09:05:55.347975 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:55.347978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:55.347991 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:55.348079 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:55.348081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:55.348123 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:55.348199 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:55.348615 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.349196 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.350371 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.350620 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.350721 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.353221 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47825: BlockingRecv error: recv got EOF from 127.0.98.1:47825 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":119,"threads_started":1}
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:55.555253 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:55.549724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:58067)
0620 09:05:55.550016 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:55.550028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:55.550173 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:55.550480 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:55.550483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:55.550683 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:55.550779 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.550784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.551546 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.551553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.552121 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.552129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.552601 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.552604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.552689 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.553314 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:55.553326 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:55.555064 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
W20260620 09:05:55.555325 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:55.549774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58067 (local address 127.0.98.62:34637)
0620 09:05:55.549944 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:55.549948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:55.550122 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:05:55.550312 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:55.550314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:55.550358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:55.550458 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:55.550914 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.551441 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.552265 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.552492 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.552598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.555199 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58067: BlockingRecv error: recv got EOF from 127.0.98.2:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:55 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:55.738345 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:55.733124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:46261)
0620 09:05:55.733377 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:55.733390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:55.733487 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:55.733717 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:55.733719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:55.733905 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:55.733994 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.733998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.734703 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.734706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.735338 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:55.735347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.735765 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.735767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:55.735832 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.736316 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:55.736326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:55.738210 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":64,"threads_started":1}
W20260620 09:05:55.738467 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:55.733204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46261 (local address 127.0.98.62:34637)
0620 09:05:55.733364 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:55.733368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:55.733463 (+ 95us) server_negotiation.cc:407] Connection header received
0620 09:05:55.733566 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:55.733568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:55.733611 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:55.733678 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:55.734086 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.734594 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.735448 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:55.735646 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:55.735717 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:55.738335 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46261: BlockingRecv error: recv got EOF from 127.0.98.3:46261 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:56.359716 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:56.354118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:34871)
0620 09:05:56.354417 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:56.354435 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:56.354558 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:56.354890 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:56.354893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:56.355092 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:56.355189 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.355195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.355943 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.355946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.356490 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.356496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.356899 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.356902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.356963 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.357638 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:56.357659 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:56.359578 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
W20260620 09:05:56.359885 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:56.354171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34871 (local address 127.0.98.62:34637)
0620 09:05:56.354420 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:56.354424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:56.354518 (+ 94us) server_negotiation.cc:407] Connection header received
0620 09:05:56.354653 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:56.354656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:56.354712 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:56.354820 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:56.355319 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.355848 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.356586 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.356806 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.356882 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.359747 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34871: BlockingRecv error: recv got EOF from 127.0.98.1:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":59,"threads_started":1}
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:56.562216 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:56.556085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:36421)
0620 09:05:56.556402 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:56.556417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:56.556568 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:56.556863 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:56.556866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:56.557159 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:56.557255 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.557261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.558090 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.558095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.558790 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.558799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.559310 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.559314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.559399 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.559986 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:56.560003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:56.562087 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
W20260620 09:05:56.562376 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:56.556158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36421 (local address 127.0.98.62:34637)
0620 09:05:56.556366 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:56.556371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:56.556510 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:05:56.556707 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:56.556710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:56.556751 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:56.556840 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:56.557370 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.557985 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.558904 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.559197 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.559307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.562250 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36421: BlockingRecv error: recv got EOF from 127.0.98.2:36421 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:56 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:56.745230 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:56.739268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:52565)
0620 09:05:56.739604 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:56.739620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:56.739765 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:56.740033 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:56.740036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:56.740254 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:56.740342 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.740347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.741168 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.741173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.741737 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:56.741744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.742192 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.742195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:56.742287 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.742852 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:56.742863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:56.745086 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
W20260620 09:05:56.745319 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:56.739450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52565 (local address 127.0.98.62:34637)
0620 09:05:56.739604 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:56.739610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:56.739707 (+ 97us) server_negotiation.cc:407] Connection header received
0620 09:05:56.739866 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:56.739870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:56.739924 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:56.740014 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:56.740473 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.740995 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.741857 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:56.742081 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:56.742190 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:56.745202 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52565: BlockingRecv error: recv got EOF from 127.0.98.3:52565 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:57.365828 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:57.360506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:47955)
0620 09:05:57.360812 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:57.360828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:57.360985 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:57.361324 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:57.361326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:57.361499 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:57.361595 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.361600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.362308 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.362311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.362882 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.362889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.363293 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.363296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.363358 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.363829 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:57.363840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:57.365714 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
W20260620 09:05:57.365978 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:57.360557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47955 (local address 127.0.98.62:34637)
0620 09:05:57.360801 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:57.360806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:57.360911 (+ 105us) server_negotiation.cc:407] Connection header received
0620 09:05:57.361145 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:57.361148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:57.361236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:57.361306 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:57.361691 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.362201 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.362975 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.363199 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.363276 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.365849 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47955: BlockingRecv error: recv got EOF from 127.0.98.1:47955 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:57.568137 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:57.563125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:38389)
0620 09:05:57.563416 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:57.563430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:57.563526 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:57.563741 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:57.563743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:57.563917 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:57.564009 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.564014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.564731 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.564734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.565379 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.565387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.565773 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.565775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.565830 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.566312 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:57.566323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:57.567999 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
W20260620 09:05:57.568218 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:57.563182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38389 (local address 127.0.98.62:34637)
0620 09:05:57.563332 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:57.563337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:57.563483 (+ 146us) server_negotiation.cc:407] Connection header received
0620 09:05:57.563590 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:57.563592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:57.563632 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:57.563702 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:57.564103 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.564644 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.565488 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.565686 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.565756 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.568127 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38389: BlockingRecv error: recv got EOF from 127.0.98.2:38389 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:57 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:57.750824 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:57.745922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:48865)
0620 09:05:57.746177 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:57.746189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:57.746283 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:57.746532 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:57.746534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:57.746710 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:57.746813 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.746819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.747514 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.747516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.748066 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:57.748073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.748475 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.748477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:57.748532 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.748980 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:57.748991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:57.750677 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260620 09:05:57.750895 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:57.745980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48865 (local address 127.0.98.62:34637)
0620 09:05:57.746115 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:57.746119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:57.746256 (+ 137us) server_negotiation.cc:407] Connection header received
0620 09:05:57.746380 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:57.746382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:57.746427 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:57.746491 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:57.746910 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.747427 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.748158 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:57.748388 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:57.748457 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:57.750805 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48865: BlockingRecv error: recv got EOF from 127.0.98.3:48865 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:58.377324 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:58.366553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:54611)
0620 09:05:58.366818 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:58.366835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:58.366949 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:58.367178 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:58.367180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:58.367356 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:58.367456 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.367460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.368160 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.368162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.368712 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.368718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.369197 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.369200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.369268 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.375349 (+ 6081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:58.375363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:58.377207 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
W20260620 09:05:58.377473 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:58.366686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54611 (local address 127.0.98.62:34637)
0620 09:05:58.366954 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:58.366957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:58.366970 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:05:58.367009 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:58.367011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:58.367050 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:58.367125 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:58.367554 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.368070 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.368811 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.369024 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.369119 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.377342 (+ 8223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54611: BlockingRecv error: recv got EOF from 127.0.98.1:54611 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:58.574615 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:58.569234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:51227)
0620 09:05:58.569555 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:58.569571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:58.569688 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:58.569967 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:58.569970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:58.570183 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:58.570274 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.570280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.571034 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.571039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.571607 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.571615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.572096 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.572100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.572187 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.572662 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:58.572673 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:58.574442 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
W20260620 09:05:58.574671 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:58.569269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51227 (local address 127.0.98.62:34637)
0620 09:05:58.569434 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:58.569439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:58.569655 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:05:58.569779 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:58.569781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:58.569836 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:58.569945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:58.570395 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.570922 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.571748 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.571985 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.572094 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.574565 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51227: BlockingRecv error: recv got EOF from 127.0.98.2:51227 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:58 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:58.757666 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:58.751848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:48281)
0620 09:05:58.752194 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:58.752217 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:58.752378 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:58.752659 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:58.752662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:58.752867 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:58.752958 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.752964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.754086 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.754089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.754843 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:58.754853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.755271 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.755274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:58.755337 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.755816 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:58.755827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:58.757549 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":87,"threads_started":1}
W20260620 09:05:58.757834 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:58.751968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48281 (local address 127.0.98.62:34637)
0620 09:05:58.752158 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:58.752162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:58.752313 (+ 151us) server_negotiation.cc:407] Connection header received
0620 09:05:58.752483 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:58.752486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:58.752536 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:58.752634 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:58.753151 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.753939 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.754955 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:58.755168 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:58.755249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:58.757675 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48281: BlockingRecv error: recv got EOF from 127.0.98.3:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:59.384052 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:59.378143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:33383)
0620 09:05:59.378405 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:59.378418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:59.378520 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:59.378793 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:59.378796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:59.379024 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:59.379154 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.379160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.380231 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.380234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.380861 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.380869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.381295 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.381298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.381363 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.381817 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:59.381829 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:59.383888 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
W20260620 09:05:59.384125 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:59.378231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33383 (local address 127.0.98.62:34637)
0620 09:05:59.378516 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:59.378522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:59.378541 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:05:59.378587 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:59.378589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:59.378638 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:59.378744 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:59.379247 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.380077 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.380963 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.381201 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.381282 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.384023 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33383: BlockingRecv error: recv got EOF from 127.0.98.1:33383 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:59.581156 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:59.575573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:60365)
0620 09:05:59.575929 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:59.575941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:59.576072 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:59.576342 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:59.576345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:59.576545 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:59.576635 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.576640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.577484 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.577488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.578070 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.578078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.578539 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.578543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.578630 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.579109 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:59.579120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:59.580978 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":100,"threads_started":1}
W20260620 09:05:59.581326 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:59.575712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60365 (local address 127.0.98.62:34637)
0620 09:05:59.575874 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:59.575880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:59.576015 (+ 135us) server_negotiation.cc:407] Connection header received
0620 09:05:59.576166 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:59.576169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:59.576219 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:59.576318 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:59.576757 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.577366 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.578207 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.578430 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.578536 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.581185 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60365: BlockingRecv error: recv got EOF from 127.0.98.2:60365 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:05:59 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:05:59.763957 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:05:59.758418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:34563)
0620 09:05:59.758720 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 09:05:59.758733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:05:59.758825 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:05:59.759058 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:05:59.759061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:05:59.759348 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:05:59.759445 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.759451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.760422 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.760426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.760999 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:05:59.761007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.761506 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.761509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:05:59.761594 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.762055 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:05:59.762066 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:05:59.763780 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
W20260620 09:05:59.764015 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:05:59.758476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34563 (local address 127.0.98.62:34637)
0620 09:05:59.758655 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 09:05:59.758660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:05:59.758803 (+ 143us) server_negotiation.cc:407] Connection header received
0620 09:05:59.758906 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:05:59.758908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:05:59.758952 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:05:59.759024 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:05:59.759568 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.760313 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.761157 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:05:59.761392 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:05:59.761504 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:05:59.763913 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34563: BlockingRecv error: recv got EOF from 127.0.98.3:34563 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:00.391867 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:00.384791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:57461)
0620 09:06:00.385090 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:00.385107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:00.385220 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:00.385743 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:00.385745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:00.385928 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:00.386024 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.386029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.387097 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.387104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.388205 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.388219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.388763 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.388768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.388891 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.389458 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:00.389478 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:00.391740 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
W20260620 09:06:00.392096 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:00.385197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57461 (local address 127.0.98.62:34637)
0620 09:06:00.385475 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:00.385482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:00.385497 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:06:00.385545 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:00.385550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:00.385613 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:00.385723 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:00.386145 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.386953 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.388342 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.388620 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.388703 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.391931 (+ 3228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57461: BlockingRecv error: recv got EOF from 127.0.98.1:57461 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:00.587481 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:00.581851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:41583)
0620 09:06:00.582161 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:00.582177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:00.582289 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:00.582514 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:00.582516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:00.582742 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:00.582863 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.582869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.583705 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.583707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.584367 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.584375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.584852 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.584855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.584986 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.585516 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:00.585526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:00.587354 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
W20260620 09:06:00.587630 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:00.581963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41583 (local address 127.0.98.62:34637)
0620 09:06:00.582122 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:00.582125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:00.582236 (+ 111us) server_negotiation.cc:407] Connection header received
0620 09:06:00.582378 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:00.582381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:00.582423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:00.582486 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:00.582991 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.583614 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.584466 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.584728 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.584806 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.587479 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41583: BlockingRecv error: recv got EOF from 127.0.98.2:41583 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:00 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:00.770153 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:00.764708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:43433)
0620 09:06:00.764974 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:00.764986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:00.765130 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:00.765477 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:00.765480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:00.765679 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:00.765770 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.765775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.766506 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.766511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.767094 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:00.767101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.767592 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.767596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:00.767679 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.768168 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:00.768179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:00.770039 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
W20260620 09:06:00.770399 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:00.764775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43433 (local address 127.0.98.62:34637)
0620 09:06:00.764905 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:00.764909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:00.765091 (+ 182us) server_negotiation.cc:407] Connection header received
0620 09:06:00.765278 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:00.765281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:00.765325 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:00.765430 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:00.765880 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.766395 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.767258 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:00.767481 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:00.767590 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:00.770175 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43433: BlockingRecv error: recv got EOF from 127.0.98.3:43433 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:01.397750 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:01.392711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:42495)
0620 09:06:01.393019 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:01.393032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:01.393178 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:01.393416 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:01.393418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:01.393590 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:01.393698 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.393704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.394409 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.394411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.394967 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.394974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.395369 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.395371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.395432 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.395903 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:01.395914 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:01.397625 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
W20260620 09:06:01.397891 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:01.392797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42495 (local address 127.0.98.62:34637)
0620 09:06:01.393031 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:01.393047 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:01.393147 (+ 100us) server_negotiation.cc:407] Connection header received
0620 09:06:01.393258 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:01.393260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:01.393303 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:01.393380 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:01.393799 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.394317 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.395059 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.395277 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.395351 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.397761 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42495: BlockingRecv error: recv got EOF from 127.0.98.1:42495 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:01.595168 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:01.588321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:50685)
0620 09:06:01.588652 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:01.588669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:01.588826 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:01.589284 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:01.589287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:01.589513 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:01.589636 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.589646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.590863 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.590867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.591433 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.591441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.591930 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.591933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.592024 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.592580 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:01.592592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:01.594997 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
W20260620 09:06:01.595225 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:01.588431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50685 (local address 127.0.98.62:34637)
0620 09:06:01.588678 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:01.588682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:01.588788 (+ 106us) server_negotiation.cc:407] Connection header received
0620 09:06:01.588933 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:01.588938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:01.588996 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:01.589151 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:01.589804 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.590719 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.591558 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.591814 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.591923 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.595118 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50685: BlockingRecv error: recv got EOF from 127.0.98.2:50685 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:01 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:01.776230 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:01.770991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:34981)
0620 09:06:01.771292 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:01.771305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:01.771410 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:01.771618 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:01.771620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:01.771790 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:01.771879 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.771884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.772560 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.772562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.773149 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:01.773156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.773694 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.773697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:01.773752 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.774390 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:01.774404 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:01.776110 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
W20260620 09:06:01.776352 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:01.771072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34981 (local address 127.0.98.62:34637)
0620 09:06:01.771224 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:01.771229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:01.771385 (+ 156us) server_negotiation.cc:407] Connection header received
0620 09:06:01.771491 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:01.771493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:01.771541 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:01.771607 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:01.771970 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.772472 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.773247 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:01.773589 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:01.773690 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:01.776235 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34981: BlockingRecv error: recv got EOF from 127.0.98.3:34981 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:02.403770 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:02.398560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:52539)
0620 09:06:02.398834 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:02.398847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:02.398932 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:02.399160 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:02.399162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:02.399339 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:02.399434 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.399439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.400151 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.400153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.400758 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.400765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.401199 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.401201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.401264 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.401751 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:02.401762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:02.403655 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
W20260620 09:06:02.403964 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:02.398724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52539 (local address 127.0.98.62:34637)
0620 09:06:02.398953 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:02.398957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:02.398968 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:06:02.399004 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:02.399006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:02.399046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:02.399124 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:02.399535 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.400052 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.400859 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.401096 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.401172 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.403787 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52539: BlockingRecv error: recv got EOF from 127.0.98.1:52539 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:02.602451 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:02.596680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:42215)
0620 09:06:02.596951 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:02.596964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:02.597117 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:02.597360 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:02.597362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:02.597555 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:02.597660 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.597665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.598514 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.598516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.599198 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.599207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.599804 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.599808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.599902 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.600446 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:02.600456 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:02.602336 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
W20260620 09:06:02.602596 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:02.596772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42215 (local address 127.0.98.62:34637)
0620 09:06:02.596911 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:02.596914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:02.597018 (+ 104us) server_negotiation.cc:407] Connection header received
0620 09:06:02.597205 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:02.597208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:02.597248 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:02.597322 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:02.597759 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.598413 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.599309 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.599658 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.599770 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.602471 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42215: BlockingRecv error: recv got EOF from 127.0.98.2:42215 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:02 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:02.782763 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:02.777062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:60327)
0620 09:06:02.777378 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:02.777394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:02.777544 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:02.777838 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:02.777841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:02.778047 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:02.778135 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.778140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.778888 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.778894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.779457 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:02.779465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.779922 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.779925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:02.780006 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.780594 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:02.780605 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:02.782604 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
W20260620 09:06:02.782994 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:02.777213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60327 (local address 127.0.98.62:34637)
0620 09:06:02.777358 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:02.777363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:02.777478 (+ 115us) server_negotiation.cc:407] Connection header received
0620 09:06:02.777650 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:02.777653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:02.777713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:02.777813 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:02.778249 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.778777 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.779587 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:02.779813 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:02.779921 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:02.782788 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60327: BlockingRecv error: recv got EOF from 127.0.98.3:60327 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:03.414958 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:03.404521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:43471)
0620 09:06:03.404790 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:03.404804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:03.404893 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:03.405132 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:03.405134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:03.405354 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:03.405483 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.405494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.406423 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.406428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.406987 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.406995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.407520 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.407524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.407616 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.412875 (+ 5259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:03.412886 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:03.414831 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260620 09:06:03.415107 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:03.404591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43471 (local address 127.0.98.62:34637)
0620 09:06:03.404843 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:03.404846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:03.404859 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:06:03.404983 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:03.404985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:03.405029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:03.405122 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:03.405603 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.406310 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.407127 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.407397 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.407509 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.414974 (+ 7465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43471: BlockingRecv error: recv got EOF from 127.0.98.1:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":61,"threads_started":1}
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:03.608188 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:03.603182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:42669)
0620 09:06:03.603457 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:03.603471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:03.603566 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:03.603792 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:03.603794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:03.603968 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:03.604059 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.604064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.604772 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.604775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.605323 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.605329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.605707 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.605709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.605776 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.606250 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:03.606260 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:03.608043 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260620 09:06:03.608269 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:03.603244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42669 (local address 127.0.98.62:34637)
0620 09:06:03.603375 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:03.603378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:03.603536 (+ 158us) server_negotiation.cc:407] Connection header received
0620 09:06:03.603644 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:03.603646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:03.603688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:03.603755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:03.604187 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.604682 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.605414 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.605621 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.605693 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.608171 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42669: BlockingRecv error: recv got EOF from 127.0.98.2:42669 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:03 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:03.792112 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:03.783833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:37211)
0620 09:06:03.784204 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:03.784218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:03.784376 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:03.784711 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:03.784715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:03.784979 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:03.785267 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.785278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.786463 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.786466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.787369 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:03.787380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.788527 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.788534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:03.788637 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.789399 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:03.789417 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:03.791910 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":77,"threads_started":1}
W20260620 09:06:03.792451 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:03.783900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37211 (local address 127.0.98.62:34637)
0620 09:06:03.784043 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:03.784048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:03.784323 (+ 275us) server_negotiation.cc:407] Connection header received
0620 09:06:03.784486 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:03.784490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:03.784554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:03.784650 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:03.785389 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.786249 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.787493 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:03.788146 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:03.788286 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:03.792148 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37211: BlockingRecv error: recv got EOF from 127.0.98.3:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:04.421461 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:04.415922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:44213)
0620 09:06:04.416264 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:04.416278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:04.416392 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:04.416666 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:04.416669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:04.416891 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:04.416980 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.416986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.417795 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.417799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.418361 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.418369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.418829 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.418832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.418922 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.419423 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:04.419434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:04.421343 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260620 09:06:04.421684 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:04.416024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44213 (local address 127.0.98.62:34637)
0620 09:06:04.416278 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:04.416283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:04.416334 (+ 51us) server_negotiation.cc:407] Connection header received
0620 09:06:04.416492 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:04.416494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:04.416537 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:04.416643 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:04.417145 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.417684 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.418480 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.418709 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.418813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.421485 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44213: BlockingRecv error: recv got EOF from 127.0.98.1:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":57,"threads_started":1}
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:04.614509 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:04.609142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:46233)
0620 09:06:04.609399 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:04.609413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:04.609506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:04.609739 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:04.609741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:04.609918 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:04.610014 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.610020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.610763 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.610765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.611347 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.611356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.611770 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.611772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.611835 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.612434 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:04.612446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:04.614337 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
W20260620 09:06:04.614856 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:04.609171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46233 (local address 127.0.98.62:34637)
0620 09:06:04.609312 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:04.609315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:04.609479 (+ 164us) server_negotiation.cc:407] Connection header received
0620 09:06:04.609586 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:04.609588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:04.609630 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:04.609700 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:04.610111 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.610625 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.611453 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.611665 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.611745 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.614498 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46233: BlockingRecv error: recv got EOF from 127.0.98.2:46233 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:04 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:04.798404 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:04.792890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:51407)
0620 09:06:04.793209 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:04.793227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:04.793362 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:04.793629 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:04.793632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:04.793850 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:04.793940 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.793945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.794689 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.794694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.795259 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:04.795266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.795730 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.795734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:04.795826 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.796383 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:04.796394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:04.798291 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
W20260620 09:06:04.798549 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:04.793063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51407 (local address 127.0.98.62:34637)
0620 09:06:04.793193 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:04.793197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:04.793311 (+ 114us) server_negotiation.cc:407] Connection header received
0620 09:06:04.793461 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:04.793466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:04.793520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:04.793606 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:04.794058 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.794580 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.795381 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:04.795609 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:04.795722 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:04.798434 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51407: BlockingRecv error: recv got EOF from 127.0.98.3:51407 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:05.427647 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:05.422324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:49125)
0620 09:06:05.422607 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:05.422621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:05.422717 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:05.422956 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:05.422958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:05.423132 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:05.423244 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.423250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.423927 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.423930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.424466 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.424472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.424942 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.424947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.425032 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.425650 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:05.425669 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:05.427529 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":112,"threads_started":1}
W20260620 09:06:05.427822 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:05.422410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49125 (local address 127.0.98.62:34637)
0620 09:06:05.422632 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:05.422635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:05.422668 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:06:05.422795 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:05.422797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:05.422839 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:05.422919 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:05.423349 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.423843 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.424554 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.424806 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.424918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.427655 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49125: BlockingRecv error: recv got EOF from 127.0.98.1:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:05.621589 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:05.615380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:49015)
0620 09:06:05.615662 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:05.615675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:05.615785 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:05.616013 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:05.616015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:05.616193 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:05.616290 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.616303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.617007 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.617009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.617580 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.617587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.617983 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.617986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.618047 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.618683 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:05.618701 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:05.621181 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
W20260620 09:06:05.621610 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:05.615429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49015 (local address 127.0.98.62:34637)
0620 09:06:05.615572 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:05.615577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:05.615754 (+ 177us) server_negotiation.cc:407] Connection header received
0620 09:06:05.615861 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:05.615863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:05.615906 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:05.615972 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:05.616395 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.616915 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.617676 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.617887 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.617962 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.621502 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49015: BlockingRecv error: recv got EOF from 127.0.98.2:49015 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:05 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:05.804396 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:05.799250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:60729)
0620 09:06:05.799531 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:05.799544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:05.799649 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:05.799901 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:05.799903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:05.800082 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:05.800175 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.800180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.800851 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.800853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.801411 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:05.801418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.801813 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.801815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:05.801872 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.802426 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:05.802446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:05.804265 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
W20260620 09:06:05.804508 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:05.799362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60729 (local address 127.0.98.62:34637)
0620 09:06:05.799516 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:05.799519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:05.799616 (+ 97us) server_negotiation.cc:407] Connection header received
0620 09:06:05.799740 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:05.799742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:05.799786 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:05.799851 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:05.800266 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.800764 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.801508 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:05.801724 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:05.801799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:05.804393 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60729: BlockingRecv error: recv got EOF from 127.0.98.3:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:06.433976 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:06.428384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:48999)
0620 09:06:06.428700 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:06.428714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:06.428800 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:06.429021 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:06.429023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:06.429212 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:06.429302 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.429307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.430008 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.430011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.430706 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.430715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.431131 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.431133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.431188 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.431733 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:06.431744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:06.433770 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
W20260620 09:06:06.434106 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:06.428497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48999 (local address 127.0.98.62:34637)
0620 09:06:06.428745 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:06.428748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:06.428761 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:06:06.428890 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:06.428892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:06.428935 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:06.429012 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:06.429393 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.429920 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.430803 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.431036 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.431119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.433949 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48999: BlockingRecv error: recv got EOF from 127.0.98.1:48999 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":59,"threads_started":1}
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:06.628760 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:06.622597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:35763)
0620 09:06:06.622869 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:06.622885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:06.622997 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:06.623289 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:06.623292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:06.623514 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:06.623636 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.623643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.624740 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.624744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.625544 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.625553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.626019 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.626022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.626117 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.626577 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:06.626588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:06.628645 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
W20260620 09:06:06.629112 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:06.622670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35763 (local address 127.0.98.62:34637)
0620 09:06:06.622863 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:06.622868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:06.622953 (+ 85us) server_negotiation.cc:407] Connection header received
0620 09:06:06.623114 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:06.623124 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:06.623175 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:06.623272 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:06.623760 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.624592 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.625666 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.625906 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.626017 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.628759 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35763: BlockingRecv error: recv got EOF from 127.0.98.2:35763 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:06 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:06.810778 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:06.805234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:44359)
0620 09:06:06.805573 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:06.805586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:06.805718 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:06.805981 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:06.805984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:06.806217 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:06.806307 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.806313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.807091 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.807095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.807661 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:06.807669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.808132 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.808135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:06.808218 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.808714 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:06.808726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:06.810606 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
W20260620 09:06:06.810817 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:06.805353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44359 (local address 127.0.98.62:34637)
0620 09:06:06.805521 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:06.805525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:06.805657 (+ 132us) server_negotiation.cc:407] Connection header received
0620 09:06:06.805813 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:06.805817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:06.805873 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:06.805962 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:06.806430 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.806954 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.807790 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:06.808017 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:06.808129 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:06.810719 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44359: BlockingRecv error: recv got EOF from 127.0.98.3:44359 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:07.439823 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:07.434805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:35145)
0620 09:06:07.435068 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:07.435081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:07.435174 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:07.435402 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:07.435404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:07.435574 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:07.435667 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.435672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.436389 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.436391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.436933 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.436940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.437368 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.437370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.437431 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.437966 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:07.437977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:07.439712 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260620 09:06:07.439994 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:07.434850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35145 (local address 127.0.98.62:34637)
0620 09:06:07.435082 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:07.435086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:07.435145 (+ 59us) server_negotiation.cc:407] Connection header received
0620 09:06:07.435251 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:07.435253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:07.435294 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:07.435368 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:07.435774 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.436282 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.437026 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.437269 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.437344 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.439828 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35145: BlockingRecv error: recv got EOF from 127.0.98.1:35145 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:07.634999 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:07.629498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:35637)
0620 09:06:07.629792 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:07.629809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:07.629921 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:07.630191 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:07.630195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:07.630401 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:07.630492 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.630497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.631242 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.631248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.631822 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.631830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.632304 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.632307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.632393 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.632923 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:07.632935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:07.634825 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
W20260620 09:06:07.635040 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:07.629644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35637 (local address 127.0.98.62:34637)
0620 09:06:07.629793 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:07.629799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:07.629864 (+ 65us) server_negotiation.cc:407] Connection header received
0620 09:06:07.630028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:07.630032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:07.630079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:07.630172 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:07.630603 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.631128 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.631940 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.632189 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.632298 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.634943 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35637: BlockingRecv error: recv got EOF from 127.0.98.2:35637 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:07 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:07.817152 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:07.811495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:45051)
0620 09:06:07.811844 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:07.811862 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:07.812009 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:07.812292 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:07.812295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:07.812502 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:07.812591 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.812597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.813509 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.813514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.814077 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:07.814085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.814560 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.814564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:07.814647 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.815148 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:07.815179 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:07.816956 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
W20260620 09:06:07.817236 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:07.811611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45051 (local address 127.0.98.62:34637)
0620 09:06:07.811819 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:07.811823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:07.811949 (+ 126us) server_negotiation.cc:407] Connection header received
0620 09:06:07.812116 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:07.812119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:07.812173 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:07.812273 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:07.812715 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.813394 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.814211 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:07.814452 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:07.814560 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:07.817099 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45051: BlockingRecv error: recv got EOF from 127.0.98.3:45051 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:08.449280 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:08.440532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:57147)
0620 09:06:08.440845 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:08.440859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:08.440949 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:08.441243 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:08.441245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:08.441420 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:08.441516 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.441521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.442271 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.442273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.442825 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.442831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.443244 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.443247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.443312 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.447086 (+ 3774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:08.447104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:08.449147 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260620 09:06:08.449445 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:08.440655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57147 (local address 127.0.98.62:34637)
0620 09:06:08.440993 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:08.440996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:08.441007 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:06:08.441072 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:08.441075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:08.441134 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:08.441208 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:08.441622 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.442182 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.442918 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.443147 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.443220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.449275 (+ 6055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57147: BlockingRecv error: recv got EOF from 127.0.98.1:57147 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:08.640766 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:08.635725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:58397)
0620 09:06:08.636092 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:08.636107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:08.636204 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:08.636425 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:08.636427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:08.636604 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:08.636703 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.636708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.637455 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.637458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.638012 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.638019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.638408 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.638410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.638469 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.638938 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:08.638949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:08.640643 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
W20260620 09:06:08.640880 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:08.635799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58397 (local address 127.0.98.62:34637)
0620 09:06:08.635959 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:08.635964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:08.636181 (+ 217us) server_negotiation.cc:407] Connection header received
0620 09:06:08.636280 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:08.636282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:08.636326 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:08.636394 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:08.636794 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.637366 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.638104 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.638318 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.638391 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.640768 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58397: BlockingRecv error: recv got EOF from 127.0.98.2:58397 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:08 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:08.823124 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:08.817968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.3:55957)
0620 09:06:08.818285 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:08.818297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:08.818391 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:08.818614 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:08.818617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:08.818791 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:08.818901 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.818906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.819617 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.819619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.820161 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:08.820168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.820586 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.820588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:08.820657 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.821201 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:08.821212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:08.822995 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
W20260620 09:06:08.823284 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:08.818017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55957 (local address 127.0.98.62:34637)
0620 09:06:08.818140 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:08.818147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:08.818362 (+ 215us) server_negotiation.cc:407] Connection header received
0620 09:06:08.818455 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:08.818457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:08.818502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:08.818573 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:08.818993 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.819519 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.820258 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:08.820491 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:08.820569 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:08.823126 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55957: BlockingRecv error: recv got EOF from 127.0.98.3:55957 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 09:06:09 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:09 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:09.455412 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:09.450210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.1:42145)
0620 09:06:09.450465 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:09.450477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:09.450591 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:09.450808 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:09.450810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:09.450978 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:09.451073 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:09.451078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.451827 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.451829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:09.452376 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:09.452383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.452770 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.452772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:09.452830 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:09.453301 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:09.453311 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:09.455254 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
W20260620 09:06:09.455559 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:09.450337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42145 (local address 127.0.98.62:34637)
0620 09:06:09.450603 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:09.450606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:09.450619 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:06:09.450653 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:09.450656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:09.450694 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:09.450771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:09.451170 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.451717 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.452471 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.452681 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.452753 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:09.455406 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42145: BlockingRecv error: recv got EOF from 127.0.98.1:42145 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
Jun 20 09:06:09 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 09:06:09 dist-test-slave-92t4 krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 09:06:09.647354 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:09.641854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.98.2:39013)
0620 09:06:09.642152 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:09.642170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:09.642296 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:09.642595 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:09.642598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:09.642825 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:09.642920 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:09.642926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.643763 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.643768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:09.644353 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:09.644374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.644824 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.644827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:09.644909 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:09.645421 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:09.645433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:09.647170 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34637: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
W20260620 09:06:09.647388 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:09.641948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39013 (local address 127.0.98.62:34637)
0620 09:06:09.642127 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:09.642132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:09.642249 (+ 117us) server_negotiation.cc:407] Connection header received
0620 09:06:09.642389 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:09.642391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:09.642435 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:09.642537 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:09.643033 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.643647 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.644478 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.644714 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.644821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:09.647285 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39013: BlockingRecv error: recv got EOF from 127.0.98.2:39013 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:06:09.723630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9143
I20260620 09:06:09.729266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9295
I20260620 09:06:09.734617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9427
I20260620 09:06:09.739751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9073
2026-06-20T09:06:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18000 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T09:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:06:09Z Disabled control of system clock
I20260620 09:06:09.771166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44987
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 09:06:09.883265 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:09.883527 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:09.883561 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:09.887499 9623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:06:09.887553 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:09.887570 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:09.887586 9623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:06:09.887601 9623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:06:09.892176 9623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:09.893224 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:09.894531 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:09.903169 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:09.903092 9623 server_base.cc:1034] running on GCE node
W20260620 09:06:09.903142 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:09.903101 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:09.903991 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:09.905295 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:09.906481 9623 hybrid_clock.cc:648] HybridClock initialized: now 1781946369906458 us; error 36 us; skew 500 ppm
I20260620 09:06:09.908274 9623 webserver.cc:460] Webserver started at http://127.0.98.62:33645/ using document root <none> and password file <none>
I20260620 09:06:09.908850 9623 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:09.908938 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:09.909231 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:09.911274 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "125985eaddef4461aa54d8bfe53c244a"
format_stamp: "Formatted at 2026-06-20 09:06:09 on dist-test-slave-92t4"
server_key: "540d1dea01e44441b41f67f05662b338"
server_key_iv: "60176d2baf3f700cc3370b285b883ef3"
server_key_version: "encryptionkey@0"
I20260620 09:06:09.911855 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "125985eaddef4461aa54d8bfe53c244a"
format_stamp: "Formatted at 2026-06-20 09:06:09 on dist-test-slave-92t4"
server_key: "540d1dea01e44441b41f67f05662b338"
server_key_iv: "60176d2baf3f700cc3370b285b883ef3"
server_key_version: "encryptionkey@0"
I20260620 09:06:09.915548 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:06:09.917956 9637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:09.919179 9623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:06:09.919322 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "125985eaddef4461aa54d8bfe53c244a"
format_stamp: "Formatted at 2026-06-20 09:06:09 on dist-test-slave-92t4"
server_key: "540d1dea01e44441b41f67f05662b338"
server_key_iv: "60176d2baf3f700cc3370b285b883ef3"
server_key_version: "encryptionkey@0"
I20260620 09:06:09.919433 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:09.938306 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:09.939028 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:09.939235 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:09.946303 9623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44987
I20260620 09:06:09.946332 9689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44987 every 8 connection(s)
I20260620 09:06:09.947268 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:06:09.950388 9690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:09.955751 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a: Bootstrap starting.
I20260620 09:06:09.957540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9623
I20260620 09:06:09.957671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:06:09.957935 392 external_mini_cluster.cc:1428] Setting key 7e2737c02bce6e6b9e354dda7c489912
I20260620 09:06:09.958281 9690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:09.959170 9690 log.cc:824] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:09.961174 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a: No bootstrap required, opened a new log
I20260620 09:06:09.964545 9690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260620 09:06:09.964865 9690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:09.964962 9690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125985eaddef4461aa54d8bfe53c244a, State: Initialized, Role: FOLLOWER
I20260620 09:06:09.965461 9690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260620 09:06:09.965615 9690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:06:09.965704 9690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:06:09.965806 9690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:09.966907 9690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260620 09:06:09.967162 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:09.959019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.98.62:44987)
0620 09:06:09.959762 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:09.959784 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:09.959827 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:06:09.960498 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:09.960514 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:09.960879 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:09.961233 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:09.961641 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.962615 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.963481 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:09.963933 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:09.964100 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:09.964993 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:09.965000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:09.965016 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:09.965523 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:09.965593 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:09.965620 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:09.965728 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:09.965918 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 09:06:09.966194 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":142,"threads_started":1}
I20260620 09:06:09.967269 9690 leader_election.cc:302] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125985eaddef4461aa54d8bfe53c244a; no voters:
I20260620 09:06:09.967574 9690 leader_election.cc:288] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:06:09.967682 9695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:09.967954 9695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [term 1 LEADER]: Becoming Leader. State: Replica: 125985eaddef4461aa54d8bfe53c244a, State: Running, Role: LEADER
I20260620 09:06:09.968252 9695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260620 09:06:09.969764 9690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:06:09.969923 9697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125985eaddef4461aa54d8bfe53c244a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } } }
I20260620 09:06:09.969883 9696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125985eaddef4461aa54d8bfe53c244a. Latest consensus state: current_term: 1 leader_uuid: "125985eaddef4461aa54d8bfe53c244a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125985eaddef4461aa54d8bfe53c244a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } } }
I20260620 09:06:09.970084 9696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:09.970041 9697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:09.970448 9699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:06:09.973560 9699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:06:09.978988 9699 catalog_manager.cc:1347] Generated new cluster ID: 9fa7e09c0e0549489a6067ffa40f23dc
I20260620 09:06:09.979063 9699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:06:10.014263 9699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:06:10.015244 9699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:06:10.024993 9699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125985eaddef4461aa54d8bfe53c244a: Generated new TSK 0
I20260620 09:06:10.025763 9699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:06:10.038170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 09:06:10.156749 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:10.157021 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:10.157121 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:10.161031 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:10.161154 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:10.161230 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:06:10.166115 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:10.167208 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:10.168546 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:10.178121 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.178074 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.178121 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.178516 9714 server_base.cc:1034] running on GCE node
I20260620 09:06:10.178867 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:10.179469 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:10.180630 9714 hybrid_clock.cc:648] HybridClock initialized: now 1781946370180615 us; error 27 us; skew 500 ppm
I20260620 09:06:10.182478 9714 webserver.cc:460] Webserver started at http://127.0.98.1:45851/ using document root <none> and password file <none>
I20260620 09:06:10.183060 9714 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:10.183120 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:10.183365 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:10.185525 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "93f4eb8496c9491baaef491e9c279c39"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "182326ddd5bb32023135f35e12c72101"
server_key_iv: "d647dab232887400e496fef3262da49c"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.186118 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93f4eb8496c9491baaef491e9c279c39"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "182326ddd5bb32023135f35e12c72101"
server_key_iv: "d647dab232887400e496fef3262da49c"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.189816 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:06:10.192426 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.193740 9714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:06:10.193859 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "93f4eb8496c9491baaef491e9c279c39"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "182326ddd5bb32023135f35e12c72101"
server_key_iv: "d647dab232887400e496fef3262da49c"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.193966 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:10.208902 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:10.209632 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:10.209830 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:10.210409 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:10.211338 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:10.211383 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.211454 9714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:10.211485 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.220045 9714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37365
I20260620 09:06:10.220085 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37365 every 8 connection(s)
I20260620 09:06:10.220942 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:06:10.225167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9714
I20260620 09:06:10.225255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:06:10.225553 392 external_mini_cluster.cc:1428] Setting key 32090cf7ff9118281b1fd97438ed0b2b
I20260620 09:06:10.227494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:06:10.231405 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.222730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42951 (local address 127.0.98.62:44987)
0620 09:06:10.222882 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:10.222887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:10.224025 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 09:06:10.225575 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:10.225579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:10.225638 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:10.225722 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:10.226776 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.227386 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.229212 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.229600 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.229721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.230822 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:10.230829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:10.230835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:10.230853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:10.230880 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:10.230893 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:10.230975 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:10.231118 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:06:10.231192 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:06:10.232220 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.223175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44987 (local address 127.0.98.1:42951)
0620 09:06:10.223839 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:10.223878 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:10.225346 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:10.225917 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:10.225924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:10.226009 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:10.226597 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.226611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.227532 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.227535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.229082 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.229093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.229755 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.229757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.229896 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.230556 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:10.230557 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:10.230596 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:10.230667 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:10.230979 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:10.230985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:10.230989 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:06:10.231085 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:06:10.231314 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260620 09:06:10.233541 9842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44987
I20260620 09:06:10.233804 9842 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:10.234325 9842 heartbeater.cc:511] Master 127.0.98.62:44987 requested a full tablet report, sending...
I20260620 09:06:10.236099 9654 ts_manager.cc:194] Registered new tserver with Master: 93f4eb8496c9491baaef491e9c279c39 (127.0.98.1:37365)
I20260620 09:06:10.237839 9654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42951
W20260620 09:06:10.339313 9845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:10.339628 9845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:10.339668 9845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:10.343657 9845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:10.343710 9845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:10.343765 9845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:06:10.348408 9845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:10.349515 9845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:10.350886 9845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:10.360354 9850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.360365 9851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.360435 9845 server_base.cc:1034] running on GCE node
W20260620 09:06:10.360365 9853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.361011 9845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:10.361701 9845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:10.362869 9845 hybrid_clock.cc:648] HybridClock initialized: now 1781946370362849 us; error 35 us; skew 500 ppm
I20260620 09:06:10.366217 9845 webserver.cc:460] Webserver started at http://127.0.98.2:41571/ using document root <none> and password file <none>
I20260620 09:06:10.366796 9845 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:10.366856 9845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:10.367044 9845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:10.369208 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "80c99b1dc46545748203c68e6d1140b1"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "249a77253ca2404ea87f21e2d43b30ea"
server_key_iv: "f99eeb0cec93612ef1f84d4f8cf196fe"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.369841 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80c99b1dc46545748203c68e6d1140b1"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "249a77253ca2404ea87f21e2d43b30ea"
server_key_iv: "f99eeb0cec93612ef1f84d4f8cf196fe"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.374037 9845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:06:10.377381 9859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.378690 9845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:06:10.378794 9845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "80c99b1dc46545748203c68e6d1140b1"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "249a77253ca2404ea87f21e2d43b30ea"
server_key_iv: "f99eeb0cec93612ef1f84d4f8cf196fe"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.378893 9845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:10.400951 9845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:10.401767 9845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:10.401924 9845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:10.402498 9845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:10.403438 9845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:10.403486 9845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.403527 9845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:10.403550 9845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.412532 9845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34059
I20260620 09:06:10.412554 9972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34059 every 8 connection(s)
I20260620 09:06:10.413617 9845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:06:10.415514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9845
I20260620 09:06:10.415627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:06:10.415989 392 external_mini_cluster.cc:1428] Setting key 0eb05d0f16886a6482550bc8fe111ac0
I20260620 09:06:10.418332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 09:06:10.425787 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.415485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41299 (local address 127.0.98.62:44987)
0620 09:06:10.415653 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:10.415659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:10.417882 (+ 2223us) server_negotiation.cc:407] Connection header received
0620 09:06:10.419889 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:10.419894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:10.419957 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:10.420048 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:10.420939 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.421919 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.423241 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.423677 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.423808 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.425314 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:10.425317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:10.425320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:10.425333 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:10.425352 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:10.425360 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:10.425421 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:10.425621 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:06:10.425673 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:06:10.427198 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.416665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44987 (local address 127.0.98.2:41299)
0620 09:06:10.417720 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:10.417774 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:10.419633 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:10.420203 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:10.420210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:10.420297 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:10.420796 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.420809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.422066 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.422069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.423116 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.423129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.423995 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.424001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.424281 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.424953 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:10.424956 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:10.425020 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:10.425152 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:10.425452 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:10.425463 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:10.425471 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:06:10.425622 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 09:06:10.425975 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":155,"threads_started":1}
I20260620 09:06:10.428460 9973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44987
I20260620 09:06:10.428706 9973 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:10.429302 9973 heartbeater.cc:511] Master 127.0.98.62:44987 requested a full tablet report, sending...
I20260620 09:06:10.430356 9654 ts_manager.cc:194] Registered new tserver with Master: 80c99b1dc46545748203c68e6d1140b1 (127.0.98.2:34059)
I20260620 09:06:10.430924 9654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41299
W20260620 09:06:10.534251 9976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:10.534566 9976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:10.534636 9976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:10.538755 9976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:10.538839 9976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:10.538928 9976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:06:10.543834 9976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:10.544981 9976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:10.546479 9976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:10.556465 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.556540 9976 server_base.cc:1034] running on GCE node
W20260620 09:06:10.556468 9981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.556459 9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.557210 9976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:10.557816 9976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:10.558966 9976 hybrid_clock.cc:648] HybridClock initialized: now 1781946370558945 us; error 60 us; skew 500 ppm
I20260620 09:06:10.560760 9976 webserver.cc:460] Webserver started at http://127.0.98.3:35309/ using document root <none> and password file <none>
I20260620 09:06:10.561399 9976 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:10.561462 9976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:10.561663 9976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:10.563681 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "da857ee694504987a85585becfbae7e2"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "6cfdf3ef6022fcd927f3cb15e7216c1c"
server_key_iv: "4377bb39f44b48cf57900581e0f36509"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.564237 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da857ee694504987a85585becfbae7e2"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "6cfdf3ef6022fcd927f3cb15e7216c1c"
server_key_iv: "4377bb39f44b48cf57900581e0f36509"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.567911 9976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 09:06:10.570482 9990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.571866 9976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:06:10.571988 9976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "da857ee694504987a85585becfbae7e2"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "6cfdf3ef6022fcd927f3cb15e7216c1c"
server_key_iv: "4377bb39f44b48cf57900581e0f36509"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.572113 9976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:10.582852 9976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:10.583523 9976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:10.583666 9976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:10.584196 9976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:10.585098 9976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:10.585181 9976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.585227 9976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:10.585249 9976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.594295 9976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43893
I20260620 09:06:10.594316 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43893 every 8 connection(s)
I20260620 09:06:10.595227 9976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:06:10.604033 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.597080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57733 (local address 127.0.98.62:44987)
0620 09:06:10.597207 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:10.597210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:10.598420 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 09:06:10.599741 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:10.599743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:10.599790 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:10.599863 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:10.600702 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.601273 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.602339 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.602541 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.602611 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.603663 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:10.603665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:10.603667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:10.603677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:10.603691 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:10.603698 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:10.603750 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:10.603876 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 09:06:10.603921 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:06:10.604889 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.597551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44987 (local address 127.0.98.3:57733)
0620 09:06:10.598235 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:10.598276 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:10.599561 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:10.600010 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:10.600016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:10.600118 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:10.600592 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.600603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.601380 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.601382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.602243 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:10.602251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:10.602656 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:10.602659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:10.602790 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:10.603445 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:10.603447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:10.603485 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:10.603546 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:10.603767 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:10.603773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:10.603776 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 09:06:10.603866 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:06:10.604098 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260620 09:06:10.605490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 9976
I20260620 09:06:10.605602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:06:10.605892 392 external_mini_cluster.cc:1428] Setting key 46d7d9c54a08d6f30dd9e13fcd0b4636
I20260620 09:06:10.606066 10104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44987
I20260620 09:06:10.606304 10104 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:10.606877 10104 heartbeater.cc:511] Master 127.0.98.62:44987 requested a full tablet report, sending...
I20260620 09:06:10.607919 9653 ts_manager.cc:194] Registered new tserver with Master: da857ee694504987a85585becfbae7e2 (127.0.98.3:43893)
I20260620 09:06:10.608491 9653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57733
I20260620 09:06:10.619781 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:06:10.622521 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:10.621727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.0.98.62:44987)
0620 09:06:10.621854 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:10.621857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:10.622031 (+ 174us) server_negotiation.cc:407] Connection header received
0620 09:06:10.622139 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:10.622141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:10.622180 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:10.622251 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:10.622421 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42598: BlockingRecv error: recv got EOF from 127.0.0.1:42598 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:06:10.622728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9714
I20260620 09:06:10.628343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9845
I20260620 09:06:10.633554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9976
I20260620 09:06:10.639299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 9623
2026-06-20T09:06:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (897 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T09:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:06:10Z Disabled control of system clock
I20260620 09:06:10.668066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35885
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:06:10.781083 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:10.781379 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:10.781417 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:10.785714 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:06:10.785780 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:10.785795 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:10.785844 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:06:10.785876 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:06:10.791472 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:10.792677 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:10.794137 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:10.802618 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.802659 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:10.802608 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:10.802883 10115 server_base.cc:1034] running on GCE node
I20260620 09:06:10.803433 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:10.804538 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:10.805722 10115 hybrid_clock.cc:648] HybridClock initialized: now 1781946370805712 us; error 30 us; skew 500 ppm
I20260620 09:06:10.807554 10115 webserver.cc:460] Webserver started at http://127.0.98.62:34073/ using document root <none> and password file <none>
I20260620 09:06:10.808151 10115 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:10.808241 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:10.808466 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:10.810560 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "7e697d87f7b74f508155522084524467"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "77966eb248f1e42d5f114d99bce22b9c"
server_key_iv: "0c512423bd032fabd2d44033276d32b0"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.811146 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e697d87f7b74f508155522084524467"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "77966eb248f1e42d5f114d99bce22b9c"
server_key_iv: "0c512423bd032fabd2d44033276d32b0"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.814826 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:06:10.817317 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:10.818536 10115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:06:10.818673 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "7e697d87f7b74f508155522084524467"
format_stamp: "Formatted at 2026-06-20 09:06:10 on dist-test-slave-92t4"
server_key: "77966eb248f1e42d5f114d99bce22b9c"
server_key_iv: "0c512423bd032fabd2d44033276d32b0"
server_key_version: "encryptionkey@0"
I20260620 09:06:10.818786 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:10.823457 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:10.824088 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:10.824270 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:10.830955 10115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35885
I20260620 09:06:10.830976 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35885 every 8 connection(s)
I20260620 09:06:10.831881 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:06:10.833607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10115
I20260620 09:06:10.833737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:06:10.834017 392 external_mini_cluster.cc:1428] Setting key 5dbc449862dbce07753b67b396c801b6
I20260620 09:06:10.835413 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:10.840096 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:10.835418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.98.62:35885)
0620 09:06:10.836119 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:10.836154 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:10.836197 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:06:10.837039 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:10.837085 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:10.837522 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:10.837880 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:10.838606 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:10.838612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:10.838627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:10.838637 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:10.838697 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:10.838721 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:10.838864 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:10.838934 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:06:10.839196 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260620 09:06:10.842154 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467: Bootstrap starting.
I20260620 09:06:10.844246 10182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:10.844913 10182 log.cc:824] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:10.846834 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467: No bootstrap required, opened a new log
I20260620 09:06:10.850286 10182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } }
I20260620 09:06:10.850518 10182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:10.850594 10182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e697d87f7b74f508155522084524467, State: Initialized, Role: FOLLOWER
I20260620 09:06:10.851058 10182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } }
I20260620 09:06:10.851200 10182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:06:10.851285 10182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:06:10.851384 10182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:10.852226 10182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } }
I20260620 09:06:10.852591 10182 leader_election.cc:302] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e697d87f7b74f508155522084524467; no voters:
I20260620 09:06:10.852926 10182 leader_election.cc:288] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:06:10.853046 10187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:10.853289 10187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [term 1 LEADER]: Becoming Leader. State: Replica: 7e697d87f7b74f508155522084524467, State: Running, Role: LEADER
I20260620 09:06:10.853621 10187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } }
I20260620 09:06:10.854164 10182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:06:10.855300 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e697d87f7b74f508155522084524467. Latest consensus state: current_term: 1 leader_uuid: "7e697d87f7b74f508155522084524467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } } }
I20260620 09:06:10.855350 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e697d87f7b74f508155522084524467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e697d87f7b74f508155522084524467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35885 } } }
I20260620 09:06:10.855420 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:10.855422 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:10.855923 10196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:06:10.858994 10196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:06:10.865034 10196 catalog_manager.cc:1347] Generated new cluster ID: 89cc972079d24d058ade042208927848
I20260620 09:06:10.865201 10196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:06:10.879372 10196 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:06:10.879689 10196 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:06:10.885865 10196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e697d87f7b74f508155522084524467: Generated new TSK 0
I20260620 09:06:10.886560 10196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:06:10.897001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:06:11.008836 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:11.009186 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:11.009234 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:11.013285 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:11.013340 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:11.013399 10206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:06:11.018194 10206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:11.019290 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:11.020686 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:11.030608 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.030655 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.030655 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:11.030898 10206 server_base.cc:1034] running on GCE node
I20260620 09:06:11.031334 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:11.031988 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:11.033115 10206 hybrid_clock.cc:648] HybridClock initialized: now 1781946371033070 us; error 50 us; skew 500 ppm
I20260620 09:06:11.034958 10206 webserver.cc:460] Webserver started at http://127.0.98.1:36613/ using document root <none> and password file <none>
I20260620 09:06:11.035547 10206 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:11.035606 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:11.035856 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:11.038302 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "6edacfb66e5940189b8f051481aa0aba"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "cbb4a3ab545cc231c521b02a31578a86"
server_key_iv: "e90dfc686274acde4bbb3222de18c72f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.038872 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6edacfb66e5940189b8f051481aa0aba"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "cbb4a3ab545cc231c521b02a31578a86"
server_key_iv: "e90dfc686274acde4bbb3222de18c72f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.042666 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 09:06:11.045032 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.046329 10206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:06:11.046460 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "6edacfb66e5940189b8f051481aa0aba"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "cbb4a3ab545cc231c521b02a31578a86"
server_key_iv: "e90dfc686274acde4bbb3222de18c72f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.046572 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:11.051226 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:11.051811 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:11.051988 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:11.052528 10206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:11.053500 10206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:11.053577 10206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.053664 10206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:11.053715 10206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.062507 10206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34535
I20260620 09:06:11.062534 10333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34535 every 8 connection(s)
I20260620 09:06:11.063417 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:06:11.069573 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.065225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47011 (local address 127.0.98.62:35885)
0620 09:06:11.065377 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:11.065381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:11.066444 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 09:06:11.067812 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:11.067816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:11.067862 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:11.067969 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:11.069078 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:11.069082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:11.069085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:11.069098 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:11.069127 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:11.069135 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:11.069245 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:11.069388 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 09:06:11.069434 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:06:11.070436 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.065643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35885 (local address 127.0.98.1:47011)
0620 09:06:11.066297 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:11.066336 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:11.067581 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:11.068141 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:11.068149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:11.068239 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:11.068796 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:11.068797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:11.068837 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:11.068904 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:11.069225 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:11.069234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:11.069239 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 09:06:11.069436 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 09:06:11.069697 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260620 09:06:11.071599 10334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35885
I20260620 09:06:11.071846 10334 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:11.072263 10334 heartbeater.cc:511] Master 127.0.98.62:35885 requested a full tablet report, sending...
I20260620 09:06:11.073299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10206
I20260620 09:06:11.073385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:06:11.073688 392 external_mini_cluster.cc:1428] Setting key e19e89817e76e81bef0b9a001b7da0ac
I20260620 09:06:11.073940 10146 ts_manager.cc:194] Registered new tserver with Master: 6edacfb66e5940189b8f051481aa0aba (127.0.98.1:34535)
I20260620 09:06:11.075766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 09:06:11.187177 10337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:11.187471 10337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:11.187518 10337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:11.191561 10337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:11.191612 10337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:11.191671 10337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:06:11.196280 10337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:11.197371 10337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:11.198717 10337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:11.208527 10342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.208520 10345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.208521 10343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:11.208892 10337 server_base.cc:1034] running on GCE node
I20260620 09:06:11.209365 10337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:11.210033 10337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:11.211225 10337 hybrid_clock.cc:648] HybridClock initialized: now 1781946371211207 us; error 30 us; skew 500 ppm
I20260620 09:06:11.213182 10337 webserver.cc:460] Webserver started at http://127.0.98.2:40079/ using document root <none> and password file <none>
I20260620 09:06:11.213773 10337 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:11.213865 10337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:11.214083 10337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:11.216208 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "f480c2bccc124612943916c50cc47019"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b4c7daab8b82da7e3ea4b86f956b06cd"
server_key_iv: "bad991e94757ac9f2beedd0576c3784f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.216779 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f480c2bccc124612943916c50cc47019"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b4c7daab8b82da7e3ea4b86f956b06cd"
server_key_iv: "bad991e94757ac9f2beedd0576c3784f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.220472 10337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:06:11.222990 10351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.224359 10337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:06:11.224490 10337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "f480c2bccc124612943916c50cc47019"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b4c7daab8b82da7e3ea4b86f956b06cd"
server_key_iv: "bad991e94757ac9f2beedd0576c3784f"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.224606 10337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:11.229243 10337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:11.229846 10337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:11.230026 10337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:11.230566 10337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:11.231455 10337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:11.231525 10337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.231591 10337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:11.231621 10337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.240211 10337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46129
I20260620 09:06:11.240229 10464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46129 every 8 connection(s)
I20260620 09:06:11.241259 10337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:06:11.242069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10337
I20260620 09:06:11.242218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:06:11.242496 392 external_mini_cluster.cc:1428] Setting key 9eedf081a1a8f054148e9245bf412ce7
I20260620 09:06:11.244465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 09:06:11.247898 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.243287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37423 (local address 127.0.98.62:35885)
0620 09:06:11.243422 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:11.243426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:11.244588 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 09:06:11.246144 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:11.246147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:11.246183 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:11.246259 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:11.247485 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:11.247487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:11.247490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:11.247500 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:11.247517 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:11.247525 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:11.247600 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:11.247760 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 09:06:11.247795 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:06:11.249236 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.243637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35885 (local address 127.0.98.2:37423)
0620 09:06:11.244447 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:11.244498 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:11.245896 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:11.246448 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:11.246460 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:11.246578 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:11.247184 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:11.247188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:11.247244 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:11.247334 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:11.247609 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:11.247620 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:11.247628 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:06:11.247799 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 09:06:11.248140 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260620 09:06:11.250444 10465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35885
I20260620 09:06:11.250725 10465 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:11.251261 10465 heartbeater.cc:511] Master 127.0.98.62:35885 requested a full tablet report, sending...
I20260620 09:06:11.252421 10146 ts_manager.cc:194] Registered new tserver with Master: f480c2bccc124612943916c50cc47019 (127.0.98.2:46129)
W20260620 09:06:11.357965 10468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:11.358215 10468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:11.358283 10468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:11.362277 10468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:11.362347 10468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:11.362448 10468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:06:11.367249 10468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:11.368314 10468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:11.369735 10468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:11.379030 10474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.379076 10476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:11.379112 10473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:11.379557 10468 server_base.cc:1034] running on GCE node
I20260620 09:06:11.379962 10468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:11.380735 10468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:11.381915 10468 hybrid_clock.cc:648] HybridClock initialized: now 1781946371381865 us; error 61 us; skew 500 ppm
I20260620 09:06:11.383844 10468 webserver.cc:460] Webserver started at http://127.0.98.3:43461/ using document root <none> and password file <none>
I20260620 09:06:11.384457 10468 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:11.384554 10468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:11.384794 10468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:11.387058 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "95afa72376b9476aac71a96ac368ec28"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b3a0cbd8fcd1039b2e2b365b32ee13d2"
server_key_iv: "a3b0d6b32978d6d85c2b9e761af11688"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.387677 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "95afa72376b9476aac71a96ac368ec28"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b3a0cbd8fcd1039b2e2b365b32ee13d2"
server_key_iv: "a3b0d6b32978d6d85c2b9e761af11688"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.391608 10468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:06:11.394145 10482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.395344 10468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:06:11.395481 10468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "95afa72376b9476aac71a96ac368ec28"
format_stamp: "Formatted at 2026-06-20 09:06:11 on dist-test-slave-92t4"
server_key: "b3a0cbd8fcd1039b2e2b365b32ee13d2"
server_key_iv: "a3b0d6b32978d6d85c2b9e761af11688"
server_key_version: "encryptionkey@0"
I20260620 09:06:11.395601 10468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:11.400208 10468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:11.400868 10468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:11.401103 10468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:11.401666 10468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:11.402583 10468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:11.402659 10468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.402730 10468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:11.402773 10468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:11.411531 10468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40787
I20260620 09:06:11.411548 10595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40787 every 8 connection(s)
I20260620 09:06:11.412456 10468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:06:11.418522 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.414274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48409 (local address 127.0.98.62:35885)
0620 09:06:11.414426 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:11.414431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:11.415621 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 09:06:11.416858 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:11.416861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:11.416897 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:11.416969 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:11.418149 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:11.418152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:11.418154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:06:11.418164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:11.418181 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:06:11.418189 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:11.418263 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:11.418380 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 09:06:11.418416 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:06:11.419430 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:11.414745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35885 (local address 127.0.98.3:48409)
0620 09:06:11.415462 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:11.415504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:11.416670 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:11.417235 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:11.417241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:11.417333 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:11.417863 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:06:11.417865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:11.417906 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:06:11.417973 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:11.418261 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:11.418267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:11.418271 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 09:06:11.418403 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 09:06:11.418627 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260620 09:06:11.420588 10596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35885
I20260620 09:06:11.420821 10596 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:11.421111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10468
I20260620 09:06:11.421228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:06:11.421339 10596 heartbeater.cc:511] Master 127.0.98.62:35885 requested a full tablet report, sending...
I20260620 09:06:11.421538 392 external_mini_cluster.cc:1428] Setting key 998ae1f2d6fb29b104011c7118c439f8
I20260620 09:06:11.422487 10145 ts_manager.cc:194] Registered new tserver with Master: 95afa72376b9476aac71a96ac368ec28 (127.0.98.3:40787)
I20260620 09:06:11.423578 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 09:06:11.426225 10185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:06:11.425475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.0.98.62:35885)
0620 09:06:11.425602 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:11.425606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:11.425772 (+ 166us) server_negotiation.cc:407] Connection header received
0620 09:06:11.425869 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:11.425871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:11.425898 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:11.425960 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:11.426134 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60652: BlockingRecv error: recv got EOF from 127.0.0.1:60652 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:06:11.426434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10206
I20260620 09:06:11.432557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10337
I20260620 09:06:11.437520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10468
I20260620 09:06:11.442869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10115
2026-06-20T09:06:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (803 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:06:11 dist-test-slave-92t4 krb5kdc[10606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:06:11 dist-test-slave-92t4 krb5kdc[10606](info): set up 2 sockets
Jun 20 09:06:11 dist-test-slave-92t4 krb5kdc[10606](info): commencing operation
krb5kdc: starting...
W20260620 09:06:12.486048 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:06:12 dist-test-slave-92t4 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:06:12.603365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:12.714947 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:12.715227 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:12.715267 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:12.719343 10619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:06:12.719403 10619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:12.719436 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:12.719452 10619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:06:12.719486 10619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:06:12.724715 10619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:12.725911 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:12.726999 10619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:12.735636 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:12.735642 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:12.735638 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:12.736038 10619 server_base.cc:1034] running on GCE node
I20260620 09:06:12.736593 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:12.737773 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:12.738929 10619 hybrid_clock.cc:648] HybridClock initialized: now 1781946372738890 us; error 54 us; skew 500 ppm
Jun 20 09:06:12 dist-test-slave-92t4 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:12.741638 10619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:06:12.742585 10619 webserver.cc:460] Webserver started at http://127.0.98.62:34323/ using document root <none> and password file <none>
I20260620 09:06:12.743155 10619 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:12.743247 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:12.743481 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:12.745754 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "e60cc3824ea241d7a0fa33cc35c19866"
format_stamp: "Formatted at 2026-06-20 09:06:12 on dist-test-slave-92t4"
server_key: "2ce995e35d26b0b058552c54e07fafbd"
server_key_iv: "f912ce48c2bc7e833b2d9163e810fecc"
server_key_version: "encryptionkey@0"
I20260620 09:06:12.746310 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "e60cc3824ea241d7a0fa33cc35c19866"
format_stamp: "Formatted at 2026-06-20 09:06:12 on dist-test-slave-92t4"
server_key: "2ce995e35d26b0b058552c54e07fafbd"
server_key_iv: "f912ce48c2bc7e833b2d9163e810fecc"
server_key_version: "encryptionkey@0"
I20260620 09:06:12.750039 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:06:12.752372 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:12.753692 10619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:06:12.753840 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "e60cc3824ea241d7a0fa33cc35c19866"
format_stamp: "Formatted at 2026-06-20 09:06:12 on dist-test-slave-92t4"
server_key: "2ce995e35d26b0b058552c54e07fafbd"
server_key_iv: "f912ce48c2bc7e833b2d9163e810fecc"
server_key_version: "encryptionkey@0"
I20260620 09:06:12.753955 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:12.774183 10619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:12.777249 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:12.777452 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:12.784513 10619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36499
I20260620 09:06:12.784797 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36499 every 8 connection(s)
I20260620 09:06:12.785606 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:06:12.788426 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:12.789466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10619
I20260620 09:06:12.789587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:06:12.789881 392 external_mini_cluster.cc:1428] Setting key 06c3bfc9770c9a9a727f067eca558597
I20260620 09:06:12.794533 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: Bootstrap starting.
I20260620 09:06:12.796927 10687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:12.797680 10687 log.cc:824] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:06:12 dist-test-slave-92t4 krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:12.799727 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: No bootstrap required, opened a new log
I20260620 09:06:12.803364 10687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } }
I20260620 09:06:12.803582 10687 raft_consensus.cc:374] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:12.803638 10687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60cc3824ea241d7a0fa33cc35c19866, State: Initialized, Role: FOLLOWER
I20260620 09:06:12.804059 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:12.791278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.98.62:36499)
0620 09:06:12.791719 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:12.791734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:12.791770 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:06:12.792408 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:12.792425 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:12.792764 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:12.793138 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:12.793923 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:12.794956 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:12.795809 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:12.796292 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:12.796458 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:12.799225 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:12.799230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:12.799243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:12.799268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:12.801138 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:12.801613 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:12.801619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:12.801624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:12.801734 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:12.802031 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:12.802035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:12.802036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:12.802432 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:12.802662 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:12.802931 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:06:12.803225 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":132,"threads_started":1}
I20260620 09:06:12.804103 10687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } }
I20260620 09:06:12.804256 10687 raft_consensus.cc:388] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:06:12.804316 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:06:12.804404 10687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:12.805392 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } }
I20260620 09:06:12.805739 10687 leader_election.cc:302] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e60cc3824ea241d7a0fa33cc35c19866; no voters:
I20260620 09:06:12.806061 10687 leader_election.cc:288] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:06:12.806159 10692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:12.806394 10692 raft_consensus.cc:686] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [term 1 LEADER]: Becoming Leader. State: Replica: e60cc3824ea241d7a0fa33cc35c19866, State: Running, Role: LEADER
I20260620 09:06:12.806695 10692 consensus_queue.cc:237] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } }
I20260620 09:06:12.807247 10687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:06:12.808357 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e60cc3824ea241d7a0fa33cc35c19866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } } }
I20260620 09:06:12.808495 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:12.808820 10693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e60cc3824ea241d7a0fa33cc35c19866. Latest consensus state: current_term: 1 leader_uuid: "e60cc3824ea241d7a0fa33cc35c19866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60cc3824ea241d7a0fa33cc35c19866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36499 } } }
I20260620 09:06:12.808920 10693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:12.808997 10701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:06:12.812042 10701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:06:12.817723 10701 catalog_manager.cc:1347] Generated new cluster ID: 352d2124bad14c31943e8f8f19bb669b
I20260620 09:06:12.817792 10701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:06:12.830056 10701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:06:12.830996 10701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:06:12.845651 10701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: Generated new TSK 0
I20260620 09:06:12.846406 10701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:06:12.876097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:12.988320 10713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:12.988601 10713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:12.988648 10713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:12.992837 10713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:12.992898 10713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:12.992998 10713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:06:12.998307 10713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:12.999425 10713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:13.000504 10713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:13.010129 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.010141 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.010166 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:13.010349 10713 server_base.cc:1034] running on GCE node
I20260620 09:06:13.010731 10713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:13.011392 10713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:13.012578 10713 hybrid_clock.cc:648] HybridClock initialized: now 1781946373012541 us; error 48 us; skew 500 ppm
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:13.015413 10713 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:06:13.016294 10713 webserver.cc:460] Webserver started at http://127.0.98.1:44903/ using document root <none> and password file <none>
I20260620 09:06:13.016863 10713 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:13.016953 10713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:13.017226 10713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:13.019285 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c8320ee1d7e47b2b3241e20939aa993"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "a5092a2165aa59486d8eb8c9c2c00853"
server_key_iv: "c382523faddc70ebd6deff62a3464168"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.019860 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c8320ee1d7e47b2b3241e20939aa993"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "a5092a2165aa59486d8eb8c9c2c00853"
server_key_iv: "c382523faddc70ebd6deff62a3464168"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.023723 10713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:06:13.026259 10728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.027568 10713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:06:13.027699 10713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "2c8320ee1d7e47b2b3241e20939aa993"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "a5092a2165aa59486d8eb8c9c2c00853"
server_key_iv: "c382523faddc70ebd6deff62a3464168"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.027810 10713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:13.057549 10713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:13.060850 10713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:13.061123 10713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:13.061750 10713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:13.062644 10713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:13.062711 10713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.062783 10713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:13.062835 10713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.071201 10713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35841
I20260620 09:06:13.071265 10841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35841 every 8 connection(s)
I20260620 09:06:13.072152 10713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:06:13.072700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10713
I20260620 09:06:13.072814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:06:13.073161 392 external_mini_cluster.cc:1428] Setting key 8f23000b4f80736247a492e3e8ea2279
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:13.091224 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.074200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33713 (local address 127.0.98.62:36499)
0620 09:06:13.074342 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.074346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.075937 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 09:06:13.077202 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.077205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.077251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.077322 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:13.078991 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.079598 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.081511 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.081730 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.081809 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.087178 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:13.087185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:13.087191 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:13.087226 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:13.088768 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.089429 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.089432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.089433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.089504 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.089906 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.089908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.089910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.090179 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:13.090258 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.090962 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.091095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:06:13.092717 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.074714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36499 (local address 127.0.98.1:33713)
0620 09:06:13.075796 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.075839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.076927 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.077530 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.077540 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.078098 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:13.078831 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.078854 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.079738 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.079745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.081385 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.081399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.081890 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.081896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.082153 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.083007 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:13.083034 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:13.086954 (+ 3920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:13.088893 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.088903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.088924 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.089325 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.089600 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.089605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.089610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.089805 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.090298 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:13.090308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:13.090685 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.090920 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.091452 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":749,"threads_started":1}
I20260620 09:06:13.094097 10842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36499
I20260620 09:06:13.094364 10842 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:13.094880 10842 heartbeater.cc:511] Master 127.0.98.62:36499 requested a full tablet report, sending...
I20260620 09:06:13.096391 10651 ts_manager.cc:194] Registered new tserver with Master: 2c8320ee1d7e47b2b3241e20939aa993 (127.0.98.1:35841)
I20260620 09:06:13.097664 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:06:13.103608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:13.214941 10847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:13.215227 10847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:13.215272 10847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:13.219321 10847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:13.219378 10847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:13.219480 10847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:06:13.224781 10847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:13.225929 10847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:13.227028 10847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:13.236433 10852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.236433 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.236439 10853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:13.236734 10847 server_base.cc:1034] running on GCE node
I20260620 09:06:13.237254 10847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:13.237906 10847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:13.239085 10847 hybrid_clock.cc:648] HybridClock initialized: now 1781946373239068 us; error 38 us; skew 500 ppm
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:13.241763 10847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:06:13.242622 10847 webserver.cc:460] Webserver started at http://127.0.98.2:36247/ using document root <none> and password file <none>
I20260620 09:06:13.243185 10847 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:13.243279 10847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:13.243512 10847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:13.245719 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "a57be3412edc45b9b110fac3bed964c1"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "b2e9859bac7bcc7768ce7459ab69d21f"
server_key_iv: "b308e63502105224ab45d874b925b2c1"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.246280 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a57be3412edc45b9b110fac3bed964c1"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "b2e9859bac7bcc7768ce7459ab69d21f"
server_key_iv: "b308e63502105224ab45d874b925b2c1"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.249989 10847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:06:13.252285 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.253557 10847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:06:13.253707 10847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "a57be3412edc45b9b110fac3bed964c1"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "b2e9859bac7bcc7768ce7459ab69d21f"
server_key_iv: "b308e63502105224ab45d874b925b2c1"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.253816 10847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:13.271351 10847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:13.274302 10847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:13.274506 10847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:13.275116 10847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:13.275920 10847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:13.275988 10847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.276057 10847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:13.276096 10847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.284777 10847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42953
I20260620 09:06:13.284797 10975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42953 every 8 connection(s)
I20260620 09:06:13.285732 10847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:06:13.289906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10847
I20260620 09:06:13.290032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:06:13.290305 392 external_mini_cluster.cc:1428] Setting key 98c3afb18651e65d42e45e738143f835
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:13.300169 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.287529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59167 (local address 127.0.98.62:36499)
0620 09:06:13.287703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.287707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.288568 (+ 861us) server_negotiation.cc:407] Connection header received
0620 09:06:13.289835 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.289839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.289901 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.290003 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:13.291349 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.291988 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.293290 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.293507 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.293589 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.296425 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:13.296429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:13.296431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:13.296454 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:13.297965 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.298556 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.298560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.298561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.298653 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.299030 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.299032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.299033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.299212 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:13.299286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.299935 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.300032 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:06:13.301190 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.287977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36499 (local address 127.0.98.2:59167)
0620 09:06:13.288419 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.288460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.289628 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.290209 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.290216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.290651 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:13.291202 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.291216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.292132 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.292136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.293170 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.293185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.293661 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.293666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.293895 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.294520 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:13.294545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:13.296258 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:13.298080 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.298088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.298103 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.298457 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.298750 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.298753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.298755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.298937 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.299321 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:13.299328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:13.299677 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.299910 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.300255 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260620 09:06:13.302429 10976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36499
I20260620 09:06:13.302675 10976 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:06:13.303243 10976 heartbeater.cc:511] Master 127.0.98.62:36499 requested a full tablet report, sending...
I20260620 09:06:13.304304 10651 ts_manager.cc:194] Registered new tserver with Master: a57be3412edc45b9b110fac3bed964c1 (127.0.98.2:42953)
I20260620 09:06:13.305183 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:06:13.317883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:13.429339 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:13.429625 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:13.429668 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:13.433944 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:13.434003 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:13.434064 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:06:13.439270 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:13.440323 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:13.441412 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:13.451069 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.451104 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:13.451104 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:13.451402 10981 server_base.cc:1034] running on GCE node
I20260620 09:06:13.451823 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:13.452450 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:13.453635 10981 hybrid_clock.cc:648] HybridClock initialized: now 1781946373453614 us; error 32 us; skew 500 ppm
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:13.459429 10981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:06:13.460289 10981 webserver.cc:460] Webserver started at http://127.0.98.3:44411/ using document root <none> and password file <none>
I20260620 09:06:13.460826 10981 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:13.460883 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:13.461110 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:06:13.463171 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "15b82e19dd9f47a49741a16b02467bc8"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "d46084c0df1783561af74643f48adc00"
server_key_iv: "48de7949176f3786e761d742e0fb531f"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.463694 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15b82e19dd9f47a49741a16b02467bc8"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "d46084c0df1783561af74643f48adc00"
server_key_iv: "48de7949176f3786e761d742e0fb531f"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.467286 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 09:06:13.469578 10996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.470781 10981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:06:13.470881 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "15b82e19dd9f47a49741a16b02467bc8"
format_stamp: "Formatted at 2026-06-20 09:06:13 on dist-test-slave-92t4"
server_key: "d46084c0df1783561af74643f48adc00"
server_key_iv: "48de7949176f3786e761d742e0fb531f"
server_key_version: "encryptionkey@0"
I20260620 09:06:13.470965 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:13.485152 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:13.487963 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:13.488113 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:13.488651 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:13.489491 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:13.489537 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.489578 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:13.489606 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:13.497956 10981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42493
I20260620 09:06:13.497993 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42493 every 8 connection(s)
I20260620 09:06:13.498858 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:06:13.504076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 10981
I20260620 09:06:13.504215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:06:13.504508 392 external_mini_cluster.cc:1428] Setting key fe4aaeeaf53da97c30dd6c69dea0f62a
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:13.512213 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.500604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48825 (local address 127.0.98.62:36499)
0620 09:06:13.500744 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.500754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.501521 (+ 767us) server_negotiation.cc:407] Connection header received
0620 09:06:13.502812 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.502815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.502875 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.502956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:13.504109 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.504780 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.506142 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.506367 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.506494 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.509129 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:13.509132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:13.509134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:13.509160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:13.510435 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.510898 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.510901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.510902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.511006 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.511288 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.511290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.511291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.511455 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:13.511530 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.511999 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.512092 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:06:13.512987 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.500929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36499 (local address 127.0.98.3:48825)
0620 09:06:13.501372 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.501412 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.502589 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.503125 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.503132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.503482 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:13.503958 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.503971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.504919 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.504923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.506035 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.506045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.506558 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.506563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.506826 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.507371 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:13.507387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:13.508932 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:13.510539 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.510544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.510557 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.510813 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.511086 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:13.511089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:13.511090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:13.511215 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:13.511540 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:13.511545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:13.511778 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.511931 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.512174 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260620 09:06:13.514148 11110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36499
I20260620 09:06:13.514401 11110 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:13.514891 11110 heartbeater.cc:511] Master 127.0.98.62:36499 requested a full tablet report, sending...
I20260620 09:06:13.515870 10651 ts_manager.cc:194] Registered new tserver with Master: 15b82e19dd9f47a49741a16b02467bc8 (127.0.98.3:42493)
I20260620 09:06:13.516446 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48825
I20260620 09:06:13.518334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:06:13.527700 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.520163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.98.62:36499)
0620 09:06:13.520277 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.520280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.520481 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:06:13.520579 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.520581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.520615 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.520683 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:13.521452 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.521943 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.523115 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.523514 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.523638 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.524857 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:13.524860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:13.524862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:13.524882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:13.526260 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.526624 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.526626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.526628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.526693 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:13.526899 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:13.526901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:13.526902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:13.527121 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:13.527223 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.527405 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.527492 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:06:13.530373 10651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:06:13.533500 10651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:06:13.547554 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.542188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.98.1:35841)
0620 09:06:13.542483 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.542499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.542659 (+ 160us) server_negotiation.cc:407] Connection header received
0620 09:06:13.542777 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.542782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.542981 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.543407 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:13.543604 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544727 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.546287 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.546857 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.546999 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547157 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.547293 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.547391 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260620 09:06:13.547880 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.541374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42953 (local address 127.0.0.1:48238)
0620 09:06:13.541986 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.542035 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.542170 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.543050 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.543056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.543078 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:13.543257 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.543263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544580 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.544583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.546103 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.546114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.547279 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.547283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.547529 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547572 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.547642 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.547746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":134,"thread_start_us":87,"threads_started":1}
I20260620 09:06:13.547907 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.541453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35841 (local address 127.0.0.1:42162)
0620 09:06:13.542549 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.542562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.542658 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.543134 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.543137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.543152 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:13.543274 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.543280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544876 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.544881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.546196 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.546204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.546995 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.546998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.547102 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547148 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.547265 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.547391 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":72,"thread_start_us":101,"threads_started":1}
I20260620 09:06:13.548317 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.541505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.0.98.2:42953)
0620 09:06:13.542519 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.542530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.542555 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:06:13.542650 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.542656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.542884 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.543041 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:13.543368 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544289 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.546226 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.546998 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.547089 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547215 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.548068 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.548181 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":128,"threads_started":1}
I20260620 09:06:13.548669 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.542526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.3:42493)
0620 09:06:13.542895 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.542907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.543106 (+ 199us) server_negotiation.cc:407] Connection header received
0620 09:06:13.543180 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.543186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.543353 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.543516 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:13.543721 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544766 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.546106 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.546944 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.547084 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547215 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.548343 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.548532 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260620 09:06:13.548980 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.542127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42493 (local address 127.0.0.1:57700)
0620 09:06:13.542917 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.542929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.543014 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.543499 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.543502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.543519 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:13.543617 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.543623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.544890 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.544892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.545989 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.545999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.547279 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.547283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.547526 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.547576 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.548788 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.548864 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":119,"thread_start_us":82,"threads_started":1}
I20260620 09:06:13.550891 10910 tablet_service.cc:1467] Processing CreateTablet for tablet 56279cf61708435a83463d73f240f5fa (DEFAULT_TABLE table=test-table [id=b44ce8232c5040029a52e602286269b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:13.550977 10776 tablet_service.cc:1467] Processing CreateTablet for tablet 56279cf61708435a83463d73f240f5fa (DEFAULT_TABLE table=test-table [id=b44ce8232c5040029a52e602286269b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:13.551431 11044 tablet_service.cc:1467] Processing CreateTablet for tablet 56279cf61708435a83463d73f240f5fa (DEFAULT_TABLE table=test-table [id=b44ce8232c5040029a52e602286269b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:13.551928 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56279cf61708435a83463d73f240f5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:13.551982 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56279cf61708435a83463d73f240f5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:13.552289 11044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56279cf61708435a83463d73f240f5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:13.558149 11127 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Bootstrap starting.
I20260620 09:06:13.559633 11129 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Bootstrap starting.
I20260620 09:06:13.560804 11127 tablet_bootstrap.cc:654] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:13.561359 11128 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Bootstrap starting.
I20260620 09:06:13.561436 11129 tablet_bootstrap.cc:654] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:13.561635 11127 log.cc:824] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:13.562635 11129 log.cc:824] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:13.563277 11128 tablet_bootstrap.cc:654] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:13.563836 11127 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: No bootstrap required, opened a new log
I20260620 09:06:13.564057 11127 ts_tablet_manager.cc:1397] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:06:13.564455 11128 log.cc:824] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:13.564670 11129 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: No bootstrap required, opened a new log
I20260620 09:06:13.564847 11129 ts_tablet_manager.cc:1397] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:06:13.566267 11128 tablet_bootstrap.cc:492] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: No bootstrap required, opened a new log
I20260620 09:06:13.566483 11128 ts_tablet_manager.cc:1397] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:06:13.567075 11127 raft_consensus.cc:348] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.567332 11127 raft_consensus.cc:374] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:13.567387 11127 raft_consensus.cc:729] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57be3412edc45b9b110fac3bed964c1, State: Initialized, Role: FOLLOWER
I20260620 09:06:13.567819 11127 consensus_queue.cc:260] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.568452 10976 heartbeater.cc:503] Master 127.0.98.62:36499 was elected leader, sending a full tablet report...
I20260620 09:06:13.568650 11129 raft_consensus.cc:348] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.568888 11129 raft_consensus.cc:374] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:13.568933 11129 raft_consensus.cc:729] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b82e19dd9f47a49741a16b02467bc8, State: Initialized, Role: FOLLOWER
I20260620 09:06:13.569258 11127 ts_tablet_manager.cc:1428] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:06:13.569377 11129 consensus_queue.cc:260] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.570325 11129 ts_tablet_manager.cc:1428] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:06:13.570257 11128 raft_consensus.cc:348] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.570501 11128 raft_consensus.cc:374] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:13.570555 11128 raft_consensus.cc:729] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8320ee1d7e47b2b3241e20939aa993, State: Initialized, Role: FOLLOWER
I20260620 09:06:13.570658 11110 heartbeater.cc:503] Master 127.0.98.62:36499 was elected leader, sending a full tablet report...
I20260620 09:06:13.570884 11128 consensus_queue.cc:260] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.571403 10842 heartbeater.cc:503] Master 127.0.98.62:36499 was elected leader, sending a full tablet report...
I20260620 09:06:13.571694 11128 ts_tablet_manager.cc:1428] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260620 09:06:13.573598 10843 tablet.cc:2369] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:06:13.720571 11134 raft_consensus.cc:482] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:06:13.720785 11134 raft_consensus.cc:504] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.722071 11134 leader_election.cc:288] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a57be3412edc45b9b110fac3bed964c1 (127.0.98.2:42953), 2c8320ee1d7e47b2b3241e20939aa993 (127.0.98.1:35841)
I20260620 09:06:13.726915 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.722441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46729 (local address 127.0.98.2:42953)
0620 09:06:13.722578 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.722584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.722606 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:06:13.722740 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.722743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.722802 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.722883 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:13.723198 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.723988 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.725594 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.726216 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.726329 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.726395 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.726712 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.726781 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:06:13.726915 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.722469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44941 (local address 127.0.98.1:35841)
0620 09:06:13.722586 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.722590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.722984 (+ 394us) server_negotiation.cc:407] Connection header received
0620 09:06:13.723050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.723052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.723094 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.723177 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:13.723403 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.724117 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.725594 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.726224 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.726329 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.726395 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.726717 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.726789 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:06:13.726924 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.722367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42953 (local address 127.0.98.3:46729)
0620 09:06:13.722480 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.722496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.722599 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.722923 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.722928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.722951 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:13.723090 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.723095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.724089 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.724091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.725448 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.725459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.726357 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.726361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.726519 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.726594 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.726675 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.726769 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":89}
I20260620 09:06:13.726933 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.722368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35841 (local address 127.0.98.3:44941)
0620 09:06:13.722844 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.722877 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:13.722962 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:13.723201 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:13.723204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:13.723219 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:13.723302 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.723306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.724214 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.724216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.725448 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:13.725459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.726357 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.726361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:13.726538 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.726597 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 09:06:13.726675 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:06:13.726769 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":222,"thread_start_us":175,"threads_started":1}
I20260620 09:06:13.727684 10930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56279cf61708435a83463d73f240f5fa" candidate_uuid: "15b82e19dd9f47a49741a16b02467bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57be3412edc45b9b110fac3bed964c1" is_pre_election: true
I20260620 09:06:13.727694 10796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56279cf61708435a83463d73f240f5fa" candidate_uuid: "15b82e19dd9f47a49741a16b02467bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8320ee1d7e47b2b3241e20939aa993" is_pre_election: true
I20260620 09:06:13.728010 10796 raft_consensus.cc:2454] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b82e19dd9f47a49741a16b02467bc8 in term 0.
I20260620 09:06:13.728011 10930 raft_consensus.cc:2454] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b82e19dd9f47a49741a16b02467bc8 in term 0.
I20260620 09:06:13.728520 10999 leader_election.cc:302] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15b82e19dd9f47a49741a16b02467bc8, 2c8320ee1d7e47b2b3241e20939aa993, a57be3412edc45b9b110fac3bed964c1; no voters:
I20260620 09:06:13.728806 11134 raft_consensus.cc:2781] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:06:13.728886 11134 raft_consensus.cc:482] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:06:13.728929 11134 raft_consensus.cc:3037] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:13.729952 11134 raft_consensus.cc:504] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.730283 11134 leader_election.cc:288] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [CANDIDATE]: Term 1 election: Requested vote from peers a57be3412edc45b9b110fac3bed964c1 (127.0.98.2:42953), 2c8320ee1d7e47b2b3241e20939aa993 (127.0.98.1:35841)
I20260620 09:06:13.730583 10930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56279cf61708435a83463d73f240f5fa" candidate_uuid: "15b82e19dd9f47a49741a16b02467bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57be3412edc45b9b110fac3bed964c1"
I20260620 09:06:13.730661 10796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56279cf61708435a83463d73f240f5fa" candidate_uuid: "15b82e19dd9f47a49741a16b02467bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8320ee1d7e47b2b3241e20939aa993"
I20260620 09:06:13.730738 10930 raft_consensus.cc:3037] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:13.730821 10796 raft_consensus.cc:3037] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:13.731815 10796 raft_consensus.cc:2454] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b82e19dd9f47a49741a16b02467bc8 in term 1.
I20260620 09:06:13.731823 10930 raft_consensus.cc:2454] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b82e19dd9f47a49741a16b02467bc8 in term 1.
I20260620 09:06:13.732092 10999 leader_election.cc:302] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b82e19dd9f47a49741a16b02467bc8, 2c8320ee1d7e47b2b3241e20939aa993; no voters:
I20260620 09:06:13.732246 11134 raft_consensus.cc:2781] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:13.732476 11134 raft_consensus.cc:686] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 15b82e19dd9f47a49741a16b02467bc8, State: Running, Role: LEADER
I20260620 09:06:13.732837 11134 consensus_queue.cc:237] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } }
I20260620 09:06:13.736311 10650 catalog_manager.cc:5495] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15b82e19dd9f47a49741a16b02467bc8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15b82e19dd9f47a49741a16b02467bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b82e19dd9f47a49741a16b02467bc8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42493 } health_report { overall_health: HEALTHY } } }
W20260620 09:06:13.749990 11111 tablet.cc:2369] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:06:13.782114 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.778862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.0.98.3:42493)
0620 09:06:13.778975 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:13.778978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:13.778991 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:06:13.779097 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:13.779099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:13.779153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:13.779239 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:06:13.779461 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.779947 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.780751 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:13.780964 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:13.781105 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:13.781363 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:06:13.781786 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:06:13.781877 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:13.781928 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 09:06:13.781977 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 09:06:13.786772 10977 tablet.cc:2369] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:06:13.788877 10796 raft_consensus.cc:1261] T 56279cf61708435a83463d73f240f5fa P 2c8320ee1d7e47b2b3241e20939aa993 [term 1 FOLLOWER]: Refusing update from remote peer 15b82e19dd9f47a49741a16b02467bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:06:13.788875 10930 raft_consensus.cc:1261] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 1 FOLLOWER]: Refusing update from remote peer 15b82e19dd9f47a49741a16b02467bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:06:13.789700 11137 consensus_queue.cc:1035] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57be3412edc45b9b110fac3bed964c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:06:13.789896 11134 consensus_queue.cc:1035] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c8320ee1d7e47b2b3241e20939aa993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:06:13.798714 11141 mvcc.cc:205] Tried to move back new op lower bound from 7298852347029938176 to 7298852346812841984. Current Snapshot: MvccSnapshot[applied={T|T < 7298852347029938176}]
I20260620 09:06:13.802368 11142 mvcc.cc:205] Tried to move back new op lower bound from 7298852347029938176 to 7298852346812841984. Current Snapshot: MvccSnapshot[applied={T|T < 7298852347029938176}]
I20260620 09:06:13.806180 10650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:06:13.806354 10650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:06:13.808911 10650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e60cc3824ea241d7a0fa33cc35c19866: Sending DeleteTablet for 3 replicas of tablet 56279cf61708435a83463d73f240f5fa
I20260620 09:06:13.809710 10776 tablet_service.cc:1514] Processing DeleteTablet for tablet 56279cf61708435a83463d73f240f5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42162
I20260620 09:06:13.809887 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet 56279cf61708435a83463d73f240f5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48238
I20260620 09:06:13.810238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10713
I20260620 09:06:13.810367 11044 tablet_service.cc:1514] Processing DeleteTablet for tablet 56279cf61708435a83463d73f240f5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57700
I20260620 09:06:13.811053 11150 tablet_replica.cc:330] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: stopping tablet replica
I20260620 09:06:13.811381 11150 raft_consensus.cc:2229] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:06:13.811705 11150 raft_consensus.cc:2258] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:06:13.812918 11151 tablet_replica.cc:330] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: stopping tablet replica
I20260620 09:06:13.813233 11150 ts_tablet_manager.cc:1905] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:06:13.813262 11151 raft_consensus.cc:2229] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:06:13.813665 11151 raft_consensus.cc:2258] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:06:13.815084 11151 ts_tablet_manager.cc:1905] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:06:13.816586 11150 ts_tablet_manager.cc:1918] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:06:13.816682 11150 log.cc:1196] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/56279cf61708435a83463d73f240f5fa
I20260620 09:06:13.816979 11150 ts_tablet_manager.cc:1939] T 56279cf61708435a83463d73f240f5fa P a57be3412edc45b9b110fac3bed964c1: Deleting consensus metadata
I20260620 09:06:13.818223 10635 catalog_manager.cc:4841] TS a57be3412edc45b9b110fac3bed964c1 (127.0.98.2:42953): tablet 56279cf61708435a83463d73f240f5fa (table test-table [id=b44ce8232c5040029a52e602286269b2]) successfully deleted
W20260620 09:06:13.818312 10637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35841 (error 108)
I20260620 09:06:13.818693 11151 ts_tablet_manager.cc:1918] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:06:13.818856 11151 log.cc:1196] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/56279cf61708435a83463d73f240f5fa
I20260620 09:06:13.819247 11151 ts_tablet_manager.cc:1939] T 56279cf61708435a83463d73f240f5fa P 15b82e19dd9f47a49741a16b02467bc8: Deleting consensus metadata
I20260620 09:06:13.819393 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:13.819042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35841 (local address 127.0.0.1:42174)
0620 09:06:13.819227 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:13.819299 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 09:06:13.819564 10637 catalog_manager.cc:4584] TS 2c8320ee1d7e47b2b3241e20939aa993 (127.0.98.1:35841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56279cf61708435a83463d73f240f5fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35841: connect: Connection refused (error 111)
I20260620 09:06:13.819995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10847
I20260620 09:06:13.820312 10635 catalog_manager.cc:4841] TS 15b82e19dd9f47a49741a16b02467bc8 (127.0.98.3:42493): tablet 56279cf61708435a83463d73f240f5fa (table test-table [id=b44ce8232c5040029a52e602286269b2]) successfully deleted
I20260620 09:06:13.826166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10981
I20260620 09:06:13.834896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 10619
2026-06-20T09:06:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2398 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[11155](info): set up 2 sockets
Jun 20 09:06:13 dist-test-slave-92t4 krb5kdc[11155](info): commencing operation
krb5kdc: starting...
W20260620 09:06:14.874131 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:06:14 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:06:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:06:15.040311 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskU_KIjO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:06:17.444 UTC [11185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:06:17.444 UTC [11185] LOG: listening on IPv4 address "127.0.98.20", port 51847
2026-06-20 09:06:17.448 UTC [11185] LOG: listening on Unix socket "/tmp/.s.PGSQL.51847"
2026-06-20 09:06:17.482 UTC [11188] LOG: database system was shut down at 2026-06-20 09:06:16 UTC
2026-06-20 09:06:17.487 UTC [11185] LOG: database system is ready to accept connections
I20260620 09:06:18.433802 392 mini_postgres.cc:96] Postgres bound to 51847
2026-06-20 09:06:18.440 UTC [11196] FATAL: database "slave" does not exist
127.0.98.20:51847 - accepting connections
I20260620 09:06:18.440825 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:06:18.455713 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:06:18.906164 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:06:18.906260 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin
2026-06-20 09:06:19,180 [I] DB FLAVOR :POSTGRES
2026-06-20 09:06:19,180 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:06:19,180 [I] Checking connection..
2026-06-20 09:06:19,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:06:19,430 [I] Checking connection passed.
2026-06-20 09:06:19,431 [I] --------- Verifying version history table ---------
2026-06-20 09:06:19,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:06:19,677 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:06:19,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:06:19,924 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:06:19,925 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:06:19,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:06:20.154 UTC [11260] WARNING: there is no transaction in progress
2026-06-20 09:06:20,160 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:06:20,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:06:20,425 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:06:20,426 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:06:20,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:06:20,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:06:20,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-92t4.c.gcp-upstream.internal','N') ;"
2026-06-20 09:06:21,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:06:21,394 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:06:21,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:06:21,627 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:06:21,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:06:21,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:06:22,148 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:06:22,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:06:23.071 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.087 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.103 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.117 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.133 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.150 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.209 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.216 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.230 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.240 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.250 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.264 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.274 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.281 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.288 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23.294 UTC [11368] WARNING: there is no transaction in progress
2026-06-20 09:06:23,904 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:06:23,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:06:24,158 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:06:24,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:06:24,396 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:06:24,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:06:24,643 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:06:24,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:06:24,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:06:25,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-92t4.c.gcp-upstream.internal';"
2026-06-20 09:06:25,343 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:06:25,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:06:25,578 [I] DB_PATCHES have already been applied
I20260620 09:06:25.584424 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskU_KIjO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:06:25.584488 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:06:25.587855 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45637
Jun 20, 2026 9:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:06:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:06:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:06:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:06:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:06:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:06:26.691190 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.250s user 0.003s sys 0.007s
Jun 20, 2026 9:06:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:06:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 9:06:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:06:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:06:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:06:48.056639 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:06:48 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946408, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:06:48.141700 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:06:48.151532 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:06:48.925654 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:06:48.925753 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms
I20260620 09:06:48.925855 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51847
2026-06-20 09:06:49,113 [I] DB FLAVOR :POSTGRES
2026-06-20 09:06:49,114 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:06:49,114 [I] Checking connection
2026-06-20 09:06:49,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:06:49,348 [I] connection success
2026-06-20 09:06:49,348 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:06:49,348 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:06:49,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:06:49,608 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:06:49,608 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:06:49,609 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:06:49,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:06:49,859 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:06:50.355422 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:06:50.548234 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:06:50.593192 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:06:50 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946408, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:50.995574 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:06:50.995736 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:06:50.995764 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:06:50.999183 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11548
I20260620 09:06:50.999333 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35229
Jun 20, 2026 9:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:06:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:06:53.259688 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.118s user 0.000s sys 0.172s
I20260620 09:06:53.259816 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:06:53.259914 392 mini_ranger_kms.cc:401] 127.0.98.20:35229/kms/v1/keys
Jun 20 09:06:53 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946413, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:06:53 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946413, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:06:54 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946413, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:06:56 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:06:56.389030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45707
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:56.502777 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:56.503055 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:56.503094 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:56.507318 11599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:06:56.507382 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:56.507402 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:56.507418 11599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:06:56.507433 11599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:06:56.513334 11599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45707
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:56.514485 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:56.515554 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:56.525166 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:56.525154 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:56.525154 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:56.525904 11599 server_base.cc:1034] running on GCE node
I20260620 09:06:56.526443 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:56.527590 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:56.528764 11599 hybrid_clock.cc:648] HybridClock initialized: now 1781946416528745 us; error 34 us; skew 500 ppm
Jun 20 09:06:56 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:56.535920 11599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:06:56.537011 11599 webserver.cc:460] Webserver started at http://127.0.98.62:46427/ using document root <none> and password file <none>
I20260620 09:06:56.537669 11599 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:56.537725 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:56.537952 11599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:06:56 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:56.585731 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "da66c35d8a91428d8d1bdf78041b9924"
format_stamp: "Formatted at 2026-06-20 09:06:56 on dist-test-slave-92t4"
server_key: "7235d94cf33caa794faef99178033b4c"
server_key_iv: "6c5c994dc89dff431149dbad7d1509e3"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:56.586355 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "da66c35d8a91428d8d1bdf78041b9924"
format_stamp: "Formatted at 2026-06-20 09:06:56 on dist-test-slave-92t4"
server_key: "7235d94cf33caa794faef99178033b4c"
server_key_iv: "6c5c994dc89dff431149dbad7d1509e3"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:56.590525 11599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:06:56.605549 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:56.607121 11599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:06:56.607275 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "da66c35d8a91428d8d1bdf78041b9924"
format_stamp: "Formatted at 2026-06-20 09:06:56 on dist-test-slave-92t4"
server_key: "7235d94cf33caa794faef99178033b4c"
server_key_iv: "6c5c994dc89dff431149dbad7d1509e3"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:56.607424 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:56.629833 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:56.632753 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:56.632984 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:56.641640 11599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45707
I20260620 09:06:56.641677 11666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45707 every 8 connection(s)
I20260620 09:06:56.642647 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:06:56.645515 11667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:56.646600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 11599
I20260620 09:06:56.646716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:06:56.647002 392 external_mini_cluster.cc:1428] Setting key 581ff366d91680536584d3bb52291166
I20260620 09:06:56.652294 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: Bootstrap starting.
I20260620 09:06:56.654865 11667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:56.655797 11667 log.cc:824] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:06:56 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:56.658054 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: No bootstrap required, opened a new log
I20260620 09:06:56.661463 11667 raft_consensus.cc:348] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } }
I20260620 09:06:56.661729 11667 raft_consensus.cc:374] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:56.661809 11667 raft_consensus.cc:729] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da66c35d8a91428d8d1bdf78041b9924, State: Initialized, Role: FOLLOWER
I20260620 09:06:56.662318 11667 consensus_queue.cc:260] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } }
I20260620 09:06:56.662470 11667 raft_consensus.cc:388] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:06:56.662582 11667 raft_consensus.cc:482] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:06:56.662781 11667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:56.663836 11667 raft_consensus.cc:504] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } }
I20260620 09:06:56.664194 11667 leader_election.cc:302] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da66c35d8a91428d8d1bdf78041b9924; no voters:
I20260620 09:06:56.664531 11667 leader_election.cc:288] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:06:56.664599 11672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:56.664683 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:56.648568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.98.62:45707)
0620 09:06:56.648999 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:56.649019 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:56.649087 (+ 68us) server_negotiation.cc:407] Connection header received
0620 09:06:56.649942 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:56.649964 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:56.650394 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:56.650762 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:56.651597 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:56.653005 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:56.654295 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:56.654980 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:56.655185 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:56.658197 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:56.658203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:56.658220 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:56.658250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:56.660956 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:56.661528 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:56.661533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:56.661539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:56.661670 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:56.661979 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:56.661983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:56.661985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:56.662555 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:56.662798 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:56.663386 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 09:06:56.663679 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260620 09:06:56.665107 11672 raft_consensus.cc:686] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [term 1 LEADER]: Becoming Leader. State: Replica: da66c35d8a91428d8d1bdf78041b9924, State: Running, Role: LEADER
I20260620 09:06:56.665463 11672 consensus_queue.cc:237] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } }
I20260620 09:06:56.665962 11667 sys_catalog.cc:564] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:06:56.666787 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da66c35d8a91428d8d1bdf78041b9924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } } }
I20260620 09:06:56.667526 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:56.666972 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da66c35d8a91428d8d1bdf78041b9924. Latest consensus state: current_term: 1 leader_uuid: "da66c35d8a91428d8d1bdf78041b9924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da66c35d8a91428d8d1bdf78041b9924" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45707 } } }
I20260620 09:06:56.667852 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924 [sys.catalog]: This master's current role is: LEADER
I20260620 09:06:56.668664 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:06:56.672282 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:06:56.672716 11667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:06:56.682626 11676 catalog_manager.cc:1347] Generated new cluster ID: 4eb689a96027409fa7088e9f52be6c6b
I20260620 09:06:56.682714 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:06:56.709251 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:06:56.711066 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:06:56.724282 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: Generated new TSK 0
I20260620 09:06:56.726225 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:06:57.383476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:57.528288 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:57.528584 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:57.528671 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:57.533394 11712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:57.533536 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:57.533658 11712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:06:57.541476 11712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:57.542865 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:57.544031 11712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:57.554867 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:57.555306 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:57.555392 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:57.556775 11712 server_base.cc:1034] running on GCE node
I20260620 09:06:57.557304 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:57.558135 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:57.559355 11712 hybrid_clock.cc:648] HybridClock initialized: now 1781946417559322 us; error 45 us; skew 500 ppm
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:57.562856 11712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:06:57.563704 11712 webserver.cc:460] Webserver started at http://127.0.98.1:42755/ using document root <none> and password file <none>
I20260620 09:06:57.564250 11712 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:57.564306 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:57.564514 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:57.580207 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "03bd4d273bc348b8bc4b605544d401e3"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "6f78a2f34133cab1fa95abee4de26c52"
server_key_iv: "15808605e0c028373ffa6c190cc4f433"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:57.581164 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03bd4d273bc348b8bc4b605544d401e3"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "6f78a2f34133cab1fa95abee4de26c52"
server_key_iv: "15808605e0c028373ffa6c190cc4f433"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:57.589928 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.003s
I20260620 09:06:57.621575 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:57.623488 11712 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260620 09:06:57.623636 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "03bd4d273bc348b8bc4b605544d401e3"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "6f78a2f34133cab1fa95abee4de26c52"
server_key_iv: "15808605e0c028373ffa6c190cc4f433"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:57.623780 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:57.649273 11712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:57.653177 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:57.653465 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:57.654294 11712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:57.655412 11712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:57.655491 11712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:57.655551 11712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:57.655617 11712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 09:06:57.668113 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46479
I20260620 09:06:57.668161 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46479 every 8 connection(s)
I20260620 09:06:57.670004 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:06:57.672611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 11712
I20260620 09:06:57.672736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:06:57.673074 392 external_mini_cluster.cc:1428] Setting key 455288d96b19e09bd0bf81c467c84678
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:57.690853 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:57.672868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45707 (local address 127.0.98.1:56829)
0620 09:06:57.675426 (+ 2558us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:57.675483 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:57.676769 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:57.677737 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:57.677746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:57.678100 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:57.678772 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:57.678790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:57.679943 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:57.679946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:57.681449 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:57.681461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:57.682259 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:57.682263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:57.682465 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:57.683313 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:57.683333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:57.685134 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:57.687485 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:57.687491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:57.687510 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:57.687873 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:57.688168 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:57.688171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:57.688173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:57.688351 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:57.688827 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:57.688833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:57.689221 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 09:06:57.689419 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 09:06:57.689780 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2325,"thread_start_us":156,"threads_started":1}
I20260620 09:06:57.691617 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:57.674018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56829 (local address 127.0.98.62:45707)
0620 09:06:57.677218 (+ 3200us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:57.677227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:57.677241 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:06:57.677279 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:57.677282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:57.677327 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:57.677409 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:57.678936 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:57.679800 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:57.681599 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:57.681967 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:57.682069 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:57.685351 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:57.685355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:57.685358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:57.685385 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:57.687354 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:57.687986 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:57.687989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:57.687991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:57.688075 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:57.688454 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:57.688457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:57.688459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:57.688695 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:57.688778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:57.691302 (+ 2524us) server_negotiation.cc:299] Negotiation successful
0620 09:06:57.691434 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3120,"thread_start_us":96,"threads_started":1}
I20260620 09:06:57.693498 11845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45707
I20260620 09:06:57.693846 11845 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:57.694542 11845 heartbeater.cc:511] Master 127.0.98.62:45707 requested a full tablet report, sending...
I20260620 09:06:57.696303 11631 ts_manager.cc:194] Registered new tserver with Master: 03bd4d273bc348b8bc4b605544d401e3 (127.0.98.1:46479)
I20260620 09:06:57.697925 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56829
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:06:57.736272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 09:06:57.906786 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:57.907254 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:57.907364 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:57.913657 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:57.913817 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:57.913969 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:06:57.922098 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:57.923843 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:57.925376 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:57.939401 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:57.940011 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:57.941102 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:57.941393 11851 server_base.cc:1034] running on GCE node
I20260620 09:06:57.942040 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:57.942870 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:57.944139 11851 hybrid_clock.cc:648] HybridClock initialized: now 1781946417944064 us; error 90 us; skew 500 ppm
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:57.948316 11851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:06:57.949947 11851 webserver.cc:460] Webserver started at http://127.0.98.2:40957/ using document root <none> and password file <none>
I20260620 09:06:57.950739 11851 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:57.950881 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:57.951182 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:06:57 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:57.976123 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "cdc6633c27ce4f44bd5c26e81267aa12"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "46ce5e7499b2f43e04f49532c950dcb6"
server_key_iv: "51d4bbb9629b59fa9b91ad3194c20f79"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:57.977110 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cdc6633c27ce4f44bd5c26e81267aa12"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "46ce5e7499b2f43e04f49532c950dcb6"
server_key_iv: "51d4bbb9629b59fa9b91ad3194c20f79"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:57.983078 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 09:06:58.008620 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.010581 11851 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260620 09:06:58.010792 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "cdc6633c27ce4f44bd5c26e81267aa12"
format_stamp: "Formatted at 2026-06-20 09:06:57 on dist-test-slave-92t4"
server_key: "46ce5e7499b2f43e04f49532c950dcb6"
server_key_iv: "51d4bbb9629b59fa9b91ad3194c20f79"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:58.011029 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:58.028323 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:58.032428 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:58.032692 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:58.033474 11851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:58.034608 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:58.034713 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.034808 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:58.034871 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.049489 11851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35603
I20260620 09:06:58.050760 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:06:58.052677 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35603 every 8 connection(s)
I20260620 09:06:58.059736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 11851
I20260620 09:06:58.059870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:06:58.060222 392 external_mini_cluster.cc:1428] Setting key 6ce4745eb398de142edebf18e37af69c
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:58.072508 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.056555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48113 (local address 127.0.98.62:45707)
0620 09:06:58.056986 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.056991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.057364 (+ 373us) server_negotiation.cc:407] Connection header received
0620 09:06:58.058689 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.058693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.058756 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.058815 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:58.061088 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.061803 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.063616 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.063965 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.064060 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.067432 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:58.067436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:58.067439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:58.067471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:58.069404 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.070082 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.070086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.070089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.070179 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.071027 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.071031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.071033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.071279 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:58.071358 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.072154 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.072304 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260620 09:06:58.074577 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.055356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45707 (local address 127.0.98.2:48113)
0620 09:06:58.057193 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.057251 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.058414 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.059329 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.059339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.059724 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:58.060666 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.060690 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.061957 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.061961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.063461 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.063474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.064270 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.064276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.064494 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.065372 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:58.065393 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:58.067224 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:58.069556 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:58.069562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:58.069581 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:58.069937 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:58.070304 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:58.070307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:58.070309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:58.070502 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:58.071446 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:58.071454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:58.071822 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.073023 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.073424 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":152,"threads_started":1}
I20260620 09:06:58.076331 11981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45707
I20260620 09:06:58.076594 11981 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:58.077337 11981 heartbeater.cc:511] Master 127.0.98.62:45707 requested a full tablet report, sending...
I20260620 09:06:58.078577 11631 ts_manager.cc:194] Registered new tserver with Master: cdc6633c27ce4f44bd5c26e81267aa12 (127.0.98.2:35603)
I20260620 09:06:58.079353 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48113
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:06:58.110188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:06:58.264135 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:06:58.264513 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:06:58.264575 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:06:58.270908 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:06:58.271045 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:06:58.271163 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:06:58.279341 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:06:58.280791 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:06:58.282199 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:06:58.293910 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:58.293910 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:06:58.294591 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:06:58.294911 11986 server_base.cc:1034] running on GCE node
I20260620 09:06:58.295341 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:06:58.296116 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:06:58.297268 11986 hybrid_clock.cc:648] HybridClock initialized: now 1781946418297235 us; error 46 us; skew 500 ppm
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:06:58.300184 11986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:06:58.301245 11986 webserver.cc:460] Webserver started at http://127.0.98.3:41309/ using document root <none> and password file <none>
I20260620 09:06:58.301867 11986 fs_manager.cc:359] Metadata directory not provided
I20260620 09:06:58.301951 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:06:58.302189 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:58.315613 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "9271d574ea98429ebee68951f107a6f7"
format_stamp: "Formatted at 2026-06-20 09:06:58 on dist-test-slave-92t4"
server_key: "92d202bcd5d849f7b1a2aa23c1b37933"
server_key_iv: "a4f7f413de2ef6b9e0d787d2d443d408"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:58.316331 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9271d574ea98429ebee68951f107a6f7"
format_stamp: "Formatted at 2026-06-20 09:06:58 on dist-test-slave-92t4"
server_key: "92d202bcd5d849f7b1a2aa23c1b37933"
server_key_iv: "a4f7f413de2ef6b9e0d787d2d443d408"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:58.322479 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 09:06:58.353845 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.355820 11986 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260620 09:06:58.355980 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "9271d574ea98429ebee68951f107a6f7"
format_stamp: "Formatted at 2026-06-20 09:06:58 on dist-test-slave-92t4"
server_key: "92d202bcd5d849f7b1a2aa23c1b37933"
server_key_iv: "a4f7f413de2ef6b9e0d787d2d443d408"
server_key_version: "kuduclusterkey@0"
I20260620 09:06:58.356153 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:06:58.382236 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:06:58.385432 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:06:58.385636 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:06:58.386273 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:06:58.387179 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:06:58.387243 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.387351 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:06:58.387391 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:06:58.398663 11986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37339
I20260620 09:06:58.398679 12114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37339 every 8 connection(s)
I20260620 09:06:58.399698 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:06:58.406184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 11986
I20260620 09:06:58.406306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:06:58.406626 392 external_mini_cluster.cc:1428] Setting key b8f82896fff263dd9b888009eb995319
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:06:58.414628 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.401575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34317 (local address 127.0.98.62:45707)
0620 09:06:58.401711 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.401716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.402664 (+ 948us) server_negotiation.cc:407] Connection header received
0620 09:06:58.403667 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.403670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.403727 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.403809 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:58.404984 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.405575 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.407262 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.407584 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.407660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.410737 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:58.410740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:58.410742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:58.410765 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:58.412366 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.412858 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.412863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.412865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.412945 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.413377 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.413381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.413383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.413620 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:58.413735 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.414341 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.414466 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:06:58.415710 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.402044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45707 (local address 127.0.98.3:34317)
0620 09:06:58.402511 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.402563 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.403448 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.404005 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.404012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.404342 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:06:58.404831 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.404856 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.405696 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.405700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.407137 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.407147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.407860 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.407865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.408513 (+ 648us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.409225 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:06:58.409249 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:06:58.410552 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:06:58.412472 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:58.412479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:58.412502 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:58.412758 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:58.413064 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:06:58.413067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:06:58.413070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:06:58.413215 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:06:58.413747 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:06:58.413754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:06:58.414068 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.414280 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.414619 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":139,"threads_started":1}
I20260620 09:06:58.416937 12115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45707
I20260620 09:06:58.417213 12115 heartbeater.cc:464] Registering TS with master...
I20260620 09:06:58.417770 12115 heartbeater.cc:511] Master 127.0.98.62:45707 requested a full tablet report, sending...
I20260620 09:06:58.418820 11631 ts_manager.cc:194] Registered new tserver with Master: 9271d574ea98429ebee68951f107a6f7 (127.0.98.3:37339)
I20260620 09:06:58.419536 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34317
I20260620 09:06:58.421149 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:06:58.432574 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.423464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.0.98.62:45707)
0620 09:06:58.423568 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.423572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.423859 (+ 287us) server_negotiation.cc:407] Connection header received
0620 09:06:58.423991 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.423994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.424049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.424120 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:06:58.424978 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.425803 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.426906 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.427250 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.427346 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.429018 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:06:58.429021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:06:58.429030 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:06:58.429078 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:06:58.430844 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.431304 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.431308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.431310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.431391 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:06:58.431706 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:06:58.431709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:06:58.431710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:06:58.431914 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:06:58.432047 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.432289 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.432405 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:06:58.436162 11631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:06:58.440296 11631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:06:58.459483 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.450791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35603 (local address 127.0.0.1:56136)
0620 09:06:58.451146 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.451191 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.451305 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.453897 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.453903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.453926 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:58.454085 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.454093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.455755 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.455758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.457129 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.457139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.458951 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.458955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.459058 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.459092 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.459207 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.459325 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260620 09:06:58.460820 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.451842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37339 (local address 127.0.0.1:60660)
0620 09:06:58.452988 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.453001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.453125 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.454240 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.454243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.454258 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:58.454336 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.454342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.455755 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.455758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.457857 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.457865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.460350 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.460354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.460498 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.460526 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.460606 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.460681 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":1337,"thread_start_us":81,"threads_started":1}
I20260620 09:06:58.461858 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.452234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.0.98.2:35603)
0620 09:06:58.453370 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.453382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.453408 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:06:58.453499 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.453505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.453728 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.454220 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:58.454402 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.455621 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.458075 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.458838 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.458954 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.459124 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.461602 (+ 2478us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.461715 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":102,"threads_started":1}
I20260620 09:06:58.462162 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.452517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.0.98.3:37339)
0620 09:06:58.453667 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.453677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.453697 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:06:58.453768 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.453776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.453943 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.454094 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:58.454455 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.455621 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.457997 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.458838 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.460137 (+ 1299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.460303 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.461299 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.461414 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":142,"threads_started":1}
I20260620 09:06:58.464885 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.452137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46479 (local address 127.0.0.1:52918)
0620 09:06:58.459867 (+ 7730us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.459885 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.459981 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.461325 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.461328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.461344 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:58.461413 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.461418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.462969 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.462972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.463748 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.463757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.464532 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.464536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.464611 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.464631 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.464695 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.464754 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7672,"thread_start_us":77,"threads_started":1}
I20260620 09:06:58.465618 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.452629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.98.1:46479)
0620 09:06:58.459336 (+ 6707us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.459343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.460730 (+ 1387us) server_negotiation.cc:407] Connection header received
0620 09:06:58.460818 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.460823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.461182 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.461563 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:58.461700 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.462845 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.463868 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.464405 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.464531 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.465404 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.465443 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.465515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6593,"thread_start_us":111,"threads_started":1}
I20260620 09:06:58.467775 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 56768707ddb0496e83e780132cfc801b (DEFAULT_TABLE table=test-table [id=f5f47114a05a46f3927afdbdec3649fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:58.467790 12035 tablet_service.cc:1467] Processing CreateTablet for tablet 56768707ddb0496e83e780132cfc801b (DEFAULT_TABLE table=test-table [id=f5f47114a05a46f3927afdbdec3649fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:58.468792 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56768707ddb0496e83e780132cfc801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:58.468798 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56768707ddb0496e83e780132cfc801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:58.469588 11914 tablet_service.cc:1467] Processing CreateTablet for tablet 56768707ddb0496e83e780132cfc801b (DEFAULT_TABLE table=test-table [id=f5f47114a05a46f3927afdbdec3649fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:06:58.470536 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56768707ddb0496e83e780132cfc801b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:06:58.477403 12133 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Bootstrap starting.
I20260620 09:06:58.480193 12133 tablet_bootstrap.cc:654] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:58.480619 12134 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: Bootstrap starting.
I20260620 09:06:58.482995 12134 tablet_bootstrap.cc:654] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:58.483865 12134 log.cc:824] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:58.484885 12133 log.cc:824] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:58.485406 12135 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Bootstrap starting.
I20260620 09:06:58.486068 12134 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: No bootstrap required, opened a new log
I20260620 09:06:58.486260 12134 ts_tablet_manager.cc:1397] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:06:58.490281 12135 tablet_bootstrap.cc:654] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Neither blocks nor log segments found. Creating new log.
I20260620 09:06:58.490774 12134 raft_consensus.cc:348] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.491043 12134 raft_consensus.cc:374] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:58.491087 12134 raft_consensus.cc:729] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bd4d273bc348b8bc4b605544d401e3, State: Initialized, Role: FOLLOWER
I20260620 09:06:58.491096 12135 log.cc:824] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Log is configured to *not* fsync() on all Append() calls
I20260620 09:06:58.491766 12134 consensus_queue.cc:260] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.492906 12134 ts_tablet_manager.cc:1428] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:06:58.493357 11845 heartbeater.cc:503] Master 127.0.98.62:45707 was elected leader, sending a full tablet report...
I20260620 09:06:58.493824 12135 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: No bootstrap required, opened a new log
I20260620 09:06:58.494045 12135 ts_tablet_manager.cc:1397] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260620 09:06:58.495190 12133 tablet_bootstrap.cc:492] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: No bootstrap required, opened a new log
I20260620 09:06:58.495362 12133 ts_tablet_manager.cc:1397] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260620 09:06:58.498167 12135 raft_consensus.cc:348] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.498430 12135 raft_consensus.cc:374] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:58.498473 12135 raft_consensus.cc:729] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc6633c27ce4f44bd5c26e81267aa12, State: Initialized, Role: FOLLOWER
I20260620 09:06:58.499256 12133 raft_consensus.cc:348] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.499498 12133 raft_consensus.cc:374] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:06:58.499552 12133 raft_consensus.cc:729] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9271d574ea98429ebee68951f107a6f7, State: Initialized, Role: FOLLOWER
I20260620 09:06:58.499469 12135 consensus_queue.cc:260] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.500195 12133 consensus_queue.cc:260] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.500523 12135 ts_tablet_manager.cc:1428] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 09:06:58.500882 11981 heartbeater.cc:503] Master 127.0.98.62:45707 was elected leader, sending a full tablet report...
I20260620 09:06:58.501425 12115 heartbeater.cc:503] Master 127.0.98.62:45707 was elected leader, sending a full tablet report...
I20260620 09:06:58.502322 12133 ts_tablet_manager.cc:1428] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:06:58.547443 12141 raft_consensus.cc:482] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:06:58.547740 12141 raft_consensus.cc:504] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.550339 12141 leader_election.cc:288] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdc6633c27ce4f44bd5c26e81267aa12 (127.0.98.2:35603), 03bd4d273bc348b8bc4b605544d401e3 (127.0.98.1:46479)
I20260620 09:06:58.554673 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.549410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35603 (local address 127.0.98.3:38629)
0620 09:06:58.549773 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.549790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.549905 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.550568 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.550571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.550592 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:58.550708 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.550714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.552123 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.552128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.553373 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.553384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.554296 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.554300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.554391 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.554412 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.554473 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.554544 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260620 09:06:58.555315 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.550087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38629 (local address 127.0.98.2:35603)
0620 09:06:58.550215 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.550220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.550235 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:06:58.550289 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.550292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.550343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.550422 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:58.550837 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.551581 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.553509 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.554071 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.554166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.554229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.555135 (+ 906us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.555207 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:06:58.555864 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56768707ddb0496e83e780132cfc801b" candidate_uuid: "9271d574ea98429ebee68951f107a6f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" is_pre_election: true
I20260620 09:06:58.556190 11933 raft_consensus.cc:2454] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9271d574ea98429ebee68951f107a6f7 in term 0.
I20260620 09:06:58.556746 12003 leader_election.cc:302] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9271d574ea98429ebee68951f107a6f7, cdc6633c27ce4f44bd5c26e81267aa12; no voters:
I20260620 09:06:58.557229 12141 raft_consensus.cc:2781] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:06:58.557386 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.550203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55941 (local address 127.0.98.1:46479)
0620 09:06:58.550310 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.550314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.552938 (+ 2624us) server_negotiation.cc:407] Connection header received
0620 09:06:58.553156 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.553160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.553213 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.553261 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:06:58.553717 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.554436 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.555966 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.556610 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.556689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.556753 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.557186 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.557266 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:06:58.557498 12141 raft_consensus.cc:482] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:06:58.557548 12141 raft_consensus.cc:3037] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:58.557926 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.549858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46479 (local address 127.0.98.3:55941)
0620 09:06:58.552845 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.552861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:06:58.553010 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:06:58.553402 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:06:58.553405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:06:58.553421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:06:58.553583 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.553589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.554632 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.554637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.555829 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:06:58.555839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.556901 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.556905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:06:58.557005 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.557028 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:06:58.557673 (+ 645us) client_negotiation.cc:240] Negotiation successful
0620 09:06:58.557752 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2912,"thread_start_us":97,"threads_started":1}
I20260620 09:06:58.558584 11798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56768707ddb0496e83e780132cfc801b" candidate_uuid: "9271d574ea98429ebee68951f107a6f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4d273bc348b8bc4b605544d401e3" is_pre_election: true
I20260620 09:06:58.558872 11798 raft_consensus.cc:2454] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9271d574ea98429ebee68951f107a6f7 in term 0.
I20260620 09:06:58.559073 12141 raft_consensus.cc:504] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
W20260620 09:06:58.559460 11982 tablet.cc:2369] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:06:58.559829 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56768707ddb0496e83e780132cfc801b" candidate_uuid: "9271d574ea98429ebee68951f107a6f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc6633c27ce4f44bd5c26e81267aa12"
I20260620 09:06:58.559942 11933 raft_consensus.cc:3037] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:58.560127 11798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56768707ddb0496e83e780132cfc801b" candidate_uuid: "9271d574ea98429ebee68951f107a6f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bd4d273bc348b8bc4b605544d401e3"
I20260620 09:06:58.560228 11798 raft_consensus.cc:3037] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:06:58.561259 11933 raft_consensus.cc:2454] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9271d574ea98429ebee68951f107a6f7 in term 1.
I20260620 09:06:58.561470 11798 raft_consensus.cc:2454] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9271d574ea98429ebee68951f107a6f7 in term 1.
I20260620 09:06:58.561726 12003 leader_election.cc:302] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9271d574ea98429ebee68951f107a6f7, cdc6633c27ce4f44bd5c26e81267aa12; no voters:
I20260620 09:06:58.562187 12141 leader_election.cc:288] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [CANDIDATE]: Term 1 election: Requested vote from peers cdc6633c27ce4f44bd5c26e81267aa12 (127.0.98.2:35603), 03bd4d273bc348b8bc4b605544d401e3 (127.0.98.1:46479)
I20260620 09:06:58.562318 12141 raft_consensus.cc:2781] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:06:58.562387 12141 raft_consensus.cc:686] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 1 LEADER]: Becoming Leader. State: Replica: 9271d574ea98429ebee68951f107a6f7, State: Running, Role: LEADER
I20260620 09:06:58.562709 12141 consensus_queue.cc:237] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } }
I20260620 09:06:58.566152 11631 catalog_manager.cc:5495] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9271d574ea98429ebee68951f107a6f7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9271d574ea98429ebee68951f107a6f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9271d574ea98429ebee68951f107a6f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20260620 09:06:58.586068 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.581121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.98.3:37339)
0620 09:06:58.581260 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:06:58.581266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:06:58.581282 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:06:58.581439 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:06:58.581443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:06:58.581510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:06:58.581614 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:06:58.581970 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.582733 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.583914 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:06:58.584238 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:06:58.584337 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:06:58.584751 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:06:58.585338 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:06:58.585450 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:06:58.585861 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 09:06:58.585921 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:06:58.595224 11798 raft_consensus.cc:1261] T 56768707ddb0496e83e780132cfc801b P 03bd4d273bc348b8bc4b605544d401e3 [term 1 FOLLOWER]: Refusing update from remote peer 9271d574ea98429ebee68951f107a6f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:06:58.596199 11933 raft_consensus.cc:1261] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 1 FOLLOWER]: Refusing update from remote peer 9271d574ea98429ebee68951f107a6f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:06:58.596971 12143 consensus_queue.cc:1035] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc6633c27ce4f44bd5c26e81267aa12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:06:58.597214 12141 consensus_queue.cc:1035] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bd4d273bc348b8bc4b605544d401e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:06:58.610162 12146 mvcc.cc:205] Tried to move back new op lower bound from 7298852530551648256 to 7298852530435842048. Current Snapshot: MvccSnapshot[applied={T|T < 7298852530551648256}]
I20260620 09:06:58.628620 11630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:06:58.628819 11630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:06:58.631765 11630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da66c35d8a91428d8d1bdf78041b9924: Sending DeleteTablet for 3 replicas of tablet 56768707ddb0496e83e780132cfc801b
I20260620 09:06:58.632797 11914 tablet_service.cc:1514] Processing DeleteTablet for tablet 56768707ddb0496e83e780132cfc801b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56136
Jun 20 09:06:58 dist-test-slave-92t4 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:06:58.633219 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 56768707ddb0496e83e780132cfc801b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52918
I20260620 09:06:58.634013 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet 56768707ddb0496e83e780132cfc801b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:06:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60660
I20260620 09:06:58.635049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 11712
I20260620 09:06:58.642894 12156 tablet_replica.cc:330] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: stopping tablet replica
I20260620 09:06:58.643258 12156 raft_consensus.cc:2229] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:06:58.643618 12156 raft_consensus.cc:2258] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:06:58.645256 12156 ts_tablet_manager.cc:1905] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:06:58.648128 12156 ts_tablet_manager.cc:1918] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:06:58.648238 12156 log.cc:1196] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/56768707ddb0496e83e780132cfc801b
I20260620 09:06:58.648563 12156 ts_tablet_manager.cc:1939] T 56768707ddb0496e83e780132cfc801b P cdc6633c27ce4f44bd5c26e81267aa12: Deleting consensus metadata
I20260620 09:06:58.650038 11616 catalog_manager.cc:4841] TS cdc6633c27ce4f44bd5c26e81267aa12 (127.0.98.2:35603): tablet 56768707ddb0496e83e780132cfc801b (table test-table [id=f5f47114a05a46f3927afdbdec3649fb]) successfully deleted
I20260620 09:06:58.668883 12158 tablet_replica.cc:330] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: stopping tablet replica
I20260620 09:06:58.671051 12158 raft_consensus.cc:2229] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:06:58.674422 12158 raft_consensus.cc:2258] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:06:58.680143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 11851
W20260620 09:06:58.680143 11615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46479 (error 108)
I20260620 09:06:58.683102 12158 ts_tablet_manager.cc:1905] T 56768707ddb0496e83e780132cfc801b P 9271d574ea98429ebee68951f107a6f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:06:58.683619 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:06:58.683284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46479 (local address 127.0.0.1:52924)
0620 09:06:58.683406 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 09:06:58.683506 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 09:06:58.683785 11615 catalog_manager.cc:4584] TS 03bd4d273bc348b8bc4b605544d401e3 (127.0.98.1:46479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56768707ddb0496e83e780132cfc801b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46479: connect: Connection refused (error 111)
I20260620 09:06:58.689189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 11986
I20260620 09:06:58.696468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 11599
I20260620 09:06:58.704978 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:06:59.039538 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:06:59.039685 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:06:59.384953 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:06:59.385 UTC [11185] LOG: received smart shutdown request
2026-06-20 09:06:59.388 UTC [11185] LOG: background worker "logical replication launcher" (PID 11194) exited with exit code 1
2026-06-20 09:06:59.389 UTC [11189] LOG: shutting down
2026-06-20 09:06:59.539 UTC [11185] LOG: database system is shut down
2026-06-20T09:06:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45819 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:06:59 dist-test-slave-92t4 krb5kdc[12169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:06:59 dist-test-slave-92t4 krb5kdc[12169](info): set up 2 sockets
Jun 20 09:06:59 dist-test-slave-92t4 krb5kdc[12169](info): commencing operation
krb5kdc: starting...
W20260620 09:07:00.697261 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:07:00 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:07:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 09:07:00.861018 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskU_KIjO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/postgres -l logfile start
2026-06-20 09:07:03.151 UTC [12199] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 09:07:03.151 UTC [12199] LOG: listening on IPv4 address "127.0.98.20", port 37859
2026-06-20 09:07:03.154 UTC [12199] LOG: listening on Unix socket "/tmp/.s.PGSQL.37859"
2026-06-20 09:07:03.174 UTC [12202] LOG: database system was shut down at 2026-06-20 09:07:01 UTC
2026-06-20 09:07:03.178 UTC [12199] LOG: database system is ready to accept connections
I20260620 09:07:04.121083 392 mini_postgres.cc:96] Postgres bound to 37859
2026-06-20 09:07:04.127 UTC [12210] FATAL: database "slave" does not exist
127.0.98.20:37859 - accepting connections
I20260620 09:07:04.127722 392 mini_ranger.cc:161] Starting Ranger...
I20260620 09:07:04.144882 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 09:07:04.591871 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 09:07:04.591986 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin
2026-06-20 09:07:04,875 [I] DB FLAVOR :POSTGRES
2026-06-20 09:07:04,876 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:07:04,876 [I] Checking connection..
2026-06-20 09:07:04,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 09:07:05,136 [I] Checking connection passed.
2026-06-20 09:07:05,136 [I] --------- Verifying version history table ---------
2026-06-20 09:07:05,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:07:05,399 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:07:05,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:07:05,660 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 09:07:05,660 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 09:07:05,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 09:07:05.916 UTC [12274] WARNING: there is no transaction in progress
2026-06-20 09:07:05,922 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 09:07:05,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 09:07:06,160 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 09:07:06,160 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:07:06,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 09:07:06,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 09:07:06,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-92t4.c.gcp-upstream.internal','N') ;"
2026-06-20 09:07:06,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:07:07,108 [I] Table x_portal_user does not exist in database ranger
2026-06-20 09:07:07,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:07:07,337 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 09:07:07,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:07:07,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:07:07,845 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 09:07:07,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 09:07:08.782 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.797 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.812 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.827 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.843 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.861 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.914 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.921 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.934 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.944 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.954 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.968 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.978 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.985 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.992 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:08.998 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 09:07:09,593 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 09:07:09,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 09:07:09,828 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 09:07:09,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 09:07:10,067 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 09:07:10,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 09:07:10,338 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 09:07:10,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 09:07:10,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 09:07:10,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-92t4.c.gcp-upstream.internal';"
2026-06-20 09:07:11,031 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 09:07:11,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 09:07:11,254 [I] DB_PATCHES have already been applied
I20260620 09:07:11.259747 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskU_KIjO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 09:07:11.259843 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 09:07:11.263139 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44427
Jun 20, 2026 9:07:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:07:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:07:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:07:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:07:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:07:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:07:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 09:07:12.371428 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.244s user 0.005s sys 0.005s
Jun 20, 2026 9:07:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:07:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 9:07:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:07:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:07:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 09:07:32.372991 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 09:07:32 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946452, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 09:07:32.456967 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 09:07:32.466535 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 09:07:33.336541 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 09:07:33.336638 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms
I20260620 09:07:33.336666 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37859
2026-06-20 09:07:33,511 [I] DB FLAVOR :POSTGRES
2026-06-20 09:07:33,511 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 09:07:33,512 [I] Checking connection
2026-06-20 09:07:33,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 09:07:33,765 [I] connection success
2026-06-20 09:07:33,765 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 09:07:33,766 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 09:07:33,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 09:07:34,046 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 09:07:34,046 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 09:07:34,046 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 09:07:34,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 09:07:34,340 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 09:07:34.790601 392 mini_ranger_kms.cc:310] Created kms service
I20260620 09:07:35.006706 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 09:07:35.051581 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 09:07:35 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946452, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:35.463267 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 09:07:35.463424 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskU_KIjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskU_KIjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 09:07:35.463446 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 09:07:35.467577 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12562
I20260620 09:07:35.467674 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51637
Jun 20, 2026 9:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:07:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:07:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:07:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 09:07:37.808827 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.352s user 0.020s sys 0.138s
I20260620 09:07:37.808981 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 09:07:37.809108 392 mini_ranger_kms.cc:401] 127.0.98.20:51637/kms/v1/keys
Jun 20 09:07:38 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 09:07:38 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:07:38 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946458, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 09:07:41 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:07:41.140935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40951
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40951
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:41.255704 12613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:41.255975 12613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:41.256026 12613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:41.260205 12613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:41.260270 12613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:41.260290 12613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:41.260306 12613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:41.260321 12613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:41.266001 12613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51637/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40951
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:41.267114 12613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:41.268198 12613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:07:41.277468 12613 server_base.cc:1034] running on GCE node
W20260620 09:07:41.277379 12619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:41.277403 12621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:41.277391 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:41.278215 12613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:41.279333 12613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:41.280501 12613 hybrid_clock.cc:648] HybridClock initialized: now 1781946461280476 us; error 38 us; skew 500 ppm
Jun 20 09:07:41 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:41.287657 12613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:41.288578 12613 webserver.cc:460] Webserver started at http://127.0.98.62:39163/ using document root <none> and password file <none>
I20260620 09:07:41.289177 12613 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:41.289247 12613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:41.289428 12613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:07:41 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:41.344086 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "dc76943db91a41958dcc6655dbc435a4"
format_stamp: "Formatted at 2026-06-20 09:07:41 on dist-test-slave-92t4"
server_key: "20421957c1f56d4882416f2818bde21b"
server_key_iv: "a29037f2bfd4f8dbba17ab00df9e8dd9"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:41.344925 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc76943db91a41958dcc6655dbc435a4"
format_stamp: "Formatted at 2026-06-20 09:07:41 on dist-test-slave-92t4"
server_key: "20421957c1f56d4882416f2818bde21b"
server_key_iv: "a29037f2bfd4f8dbba17ab00df9e8dd9"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:41.349699 12613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:07:41.364600 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:41.366194 12613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:41.366371 12613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "dc76943db91a41958dcc6655dbc435a4"
format_stamp: "Formatted at 2026-06-20 09:07:41 on dist-test-slave-92t4"
server_key: "20421957c1f56d4882416f2818bde21b"
server_key_iv: "a29037f2bfd4f8dbba17ab00df9e8dd9"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:41.366535 12613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:41.386464 12613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:41.389771 12613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:41.389981 12613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:41.397272 12613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40951
I20260620 09:07:41.397308 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40951 every 8 connection(s)
I20260620 09:07:41.398245 12613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:41.398726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 12613
I20260620 09:07:41.398882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:07:41.399166 392 external_mini_cluster.cc:1428] Setting key 0a68337debdf4762a86b45023297c831
I20260620 09:07:41.401698 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:41.408365 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Bootstrap starting.
Jun 20 09:07:41 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:41.411301 12681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:41.412681 12681 log.cc:824] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:41.414767 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:41.400547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.0.98.62:40951)
0620 09:07:41.400967 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:41.400986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:41.401030 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:07:41.401919 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:41.401942 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:41.402377 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:41.402765 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:41.403460 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:41.404991 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:41.405906 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:41.406604 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:41.406797 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:41.409634 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:41.409644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:41.409685 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:41.409720 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:41.411606 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:41.412052 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:41.412057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:41.412063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:41.412191 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:41.412461 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:41.412467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:41.412469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:41.413008 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:41.413216 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:41.413517 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 09:07:41.413835 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260620 09:07:41.415038 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: No bootstrap required, opened a new log
I20260620 09:07:41.418277 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:41.418491 12681 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:41.418531 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc76943db91a41958dcc6655dbc435a4, State: Initialized, Role: FOLLOWER
I20260620 09:07:41.418884 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:41.418989 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:41.419042 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:41.419111 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:41.420136 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:41.420437 12681 leader_election.cc:302] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc76943db91a41958dcc6655dbc435a4; no voters:
I20260620 09:07:41.420696 12681 leader_election.cc:288] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:07:41.420850 12686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:41.421097 12686 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 LEADER]: Becoming Leader. State: Replica: dc76943db91a41958dcc6655dbc435a4, State: Running, Role: LEADER
I20260620 09:07:41.421399 12686 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:41.421957 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:41.422968 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc76943db91a41958dcc6655dbc435a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } } }
I20260620 09:07:41.423115 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:41.423425 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc76943db91a41958dcc6655dbc435a4. Latest consensus state: current_term: 1 leader_uuid: "dc76943db91a41958dcc6655dbc435a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } } }
I20260620 09:07:41.423485 12695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:41.423527 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:41.426482 12695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:41.426904 12681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:07:41.436510 12695 catalog_manager.cc:1347] Generated new cluster ID: 6b1f68e2eef2420eb860a37e794e2575
I20260620 09:07:41.436681 12695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:41.470715 12695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:07:41.472154 12695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:41.483039 12695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Generated new TSK 0
I20260620 09:07:41.483956 12695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:07:42.151880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:42.290719 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:42.290968 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:42.291011 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:42.296237 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:42.296342 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:42.296452 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:07:42.302034 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:42.303185 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:42.304289 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:42.313993 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:42.314002 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:42.313952 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:42.314492 12726 server_base.cc:1034] running on GCE node
I20260620 09:07:42.314980 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:42.315634 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:42.316849 12726 hybrid_clock.cc:648] HybridClock initialized: now 1781946462316814 us; error 54 us; skew 500 ppm
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:42.319784 12726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:07:42.320670 12726 webserver.cc:460] Webserver started at http://127.0.98.1:43289/ using document root <none> and password file <none>
I20260620 09:07:42.321379 12726 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:42.321446 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:42.321687 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:42.340912 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "ff2535abc21bbeb11fd69939e7fa4bf9"
server_key_iv: "94276ae18d9ad8d4f7a661a8ff106bc2"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.341830 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "ff2535abc21bbeb11fd69939e7fa4bf9"
server_key_iv: "94276ae18d9ad8d4f7a661a8ff106bc2"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.349463 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 09:07:42.371696 12744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.373584 12726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 09:07:42.373745 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "ff2535abc21bbeb11fd69939e7fa4bf9"
server_key_iv: "94276ae18d9ad8d4f7a661a8ff106bc2"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.373906 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:42.427939 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:42.431952 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:42.432174 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:42.432973 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:42.434139 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:42.434203 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.434269 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:42.434293 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.445487 12726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46419
I20260620 09:07:42.446705 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:07:42.447530 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46419 every 8 connection(s)
I20260620 09:07:42.451264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 12726
I20260620 09:07:42.451397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:07:42.451735 392 external_mini_cluster.cc:1428] Setting key d50f1f81e831949b35fcb313cdd061d3
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:42.472440 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:42.450326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.1:53559)
0620 09:07:42.451955 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:42.452015 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:42.456270 (+ 4255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:42.458532 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:42.458541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:42.458912 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:42.459628 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:42.459648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.460932 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.460935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:42.462371 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:42.462382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.463055 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.463059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:42.463264 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:42.464034 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:42.464056 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:42.465980 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:42.468446 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:42.468453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:42.468472 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:42.468855 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:42.469240 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:42.469243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:42.469245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:42.469434 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:42.469897 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:42.469904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:42.470306 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 09:07:42.470570 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 09:07:42.470993 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":156,"threads_started":1}
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:42.474615 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:42.449827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53559 (local address 127.0.98.62:40951)
0620 09:07:42.458086 (+ 8259us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:42.458096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:42.458112 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:07:42.458153 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:42.458155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:42.458202 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:42.459780 (+ 1578us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:42.459900 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.460786 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.462524 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.462881 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.466173 (+ 3292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:42.466323 (+ 150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:42.466326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:42.466329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:42.466348 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:42.468308 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:42.468999 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:42.469002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:42.469004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:42.469135 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:42.469534 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:42.469537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:42.469538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:42.469762 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:42.469847 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:42.473652 (+ 3805us) server_negotiation.cc:299] Negotiation successful
0620 09:07:42.474404 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8168,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:07:42.476434 12859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:42.476724 12859 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:42.477380 12859 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:07:42.479234 12645 ts_manager.cc:194] Registered new tserver with Master: 893fc5a75a3c4e1ea9c2b617b5aba4e7 (127.0.98.1:46419)
I20260620 09:07:42.480868 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:07:42.521250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:42.671569 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:42.672014 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:42.672147 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:42.678340 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:42.678507 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:42.678646 12865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:07:42.686594 12865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:42.688225 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:42.689677 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:42.702719 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:42.703652 12874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:42.703938 12865 server_base.cc:1034] running on GCE node
W20260620 09:07:42.705495 12871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:42.706131 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:42.706949 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:42.722462 12865 hybrid_clock.cc:648] HybridClock initialized: now 1781946462722351 us; error 137 us; skew 500 ppm
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:42.726737 12865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:07:42.727998 12865 webserver.cc:460] Webserver started at http://127.0.98.2:32971/ using document root <none> and password file <none>
I20260620 09:07:42.728792 12865 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:42.728940 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:42.729259 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:42.753907 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "85bc6ef03df74ce6bce8a71d627a8a77"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "2fdc66863751b3dd64d35b825a6c104e"
server_key_iv: "e3a876f9f1534e61a27f9bf7a9494f56"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.754601 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85bc6ef03df74ce6bce8a71d627a8a77"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "2fdc66863751b3dd64d35b825a6c104e"
server_key_iv: "e3a876f9f1534e61a27f9bf7a9494f56"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.758773 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:07:42.775964 12881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.777583 12865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:42.777733 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "85bc6ef03df74ce6bce8a71d627a8a77"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "2fdc66863751b3dd64d35b825a6c104e"
server_key_iv: "e3a876f9f1534e61a27f9bf7a9494f56"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:42.777891 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:42.820322 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:42.824097 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:42.824308 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:42.825100 12865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:42.826232 12865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:42.826303 12865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.826380 12865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:42.826413 12865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:42.839267 12865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42743
I20260620 09:07:42.840641 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:07:42.848088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 12865
I20260620 09:07:42.848229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:07:42.848558 392 external_mini_cluster.cc:1428] Setting key 05f64cac1d7b99f74ef971a870463a64
I20260620 09:07:42.853194 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42743 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:42.891001 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:42.856634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.2:53015)
0620 09:07:42.861381 (+ 4747us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:42.861450 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:42.862680 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:42.870049 (+ 7369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:42.870060 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:42.870429 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:42.871270 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:42.871296 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.872486 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.872492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:42.874081 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:42.874093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.874966 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.874972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:42.875227 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:42.875997 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:42.876025 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:42.884889 (+ 8864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:42.887356 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:42.887364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:42.887386 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:42.887778 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:42.888105 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:42.888108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:42.888111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:42.888404 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:42.888900 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:42.888907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:42.889311 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 09:07:42.889521 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 09:07:42.889890 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4480,"thread_start_us":182,"threads_started":1}
I20260620 09:07:42.891893 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:42.857843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53015 (local address 127.0.98.62:40951)
0620 09:07:42.857975 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:42.857980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:42.861587 (+ 3607us) server_negotiation.cc:407] Connection header received
0620 09:07:42.869234 (+ 7647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:42.869240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:42.869323 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:42.869392 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:42.871473 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.872311 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.874250 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:42.874644 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:42.874728 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:42.885292 (+ 10564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:42.885301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:42.885306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:42.885350 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:42.887200 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:42.887885 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:42.887889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:42.887891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:42.887990 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:42.888510 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:42.888514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:42.888516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:42.888758 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:42.888846 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:42.891602 (+ 2756us) server_negotiation.cc:299] Negotiation successful
0620 09:07:42.891746 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:07:42.893160 12995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:42.893428 12995 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:42.894047 12995 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:07:42.895541 12645 ts_manager.cc:194] Registered new tserver with Master: 85bc6ef03df74ce6bce8a71d627a8a77 (127.0.98.2:42743)
I20260620 09:07:42.896353 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53015
I20260620 09:07:42.897284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:07:42 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 09:07:43.079958 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:43.080389 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:43.080456 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:43.085436 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:43.085525 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:43.085640 13000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:07:43.091361 13000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:43.092600 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:43.093868 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:43.103659 13005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.103659 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.104337 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.104846 13000 server_base.cc:1034] running on GCE node
I20260620 09:07:43.105346 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:43.106051 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:43.107227 13000 hybrid_clock.cc:648] HybridClock initialized: now 1781946463107185 us; error 53 us; skew 500 ppm
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:43.110066 13000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:07:43.111004 13000 webserver.cc:460] Webserver started at http://127.0.98.3:41723/ using document root <none> and password file <none>
I20260620 09:07:43.111683 13000 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:43.111774 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:43.111987 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:43.129194 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "db777e9811854a9ca6486e988108f49b"
format_stamp: "Formatted at 2026-06-20 09:07:43 on dist-test-slave-92t4"
server_key: "ea56e62b2ad8af4f0cf24c40779d1076"
server_key_iv: "61f425f36ad9dded813a958bdeb2fa77"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.130072 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "db777e9811854a9ca6486e988108f49b"
format_stamp: "Formatted at 2026-06-20 09:07:43 on dist-test-slave-92t4"
server_key: "ea56e62b2ad8af4f0cf24c40779d1076"
server_key_iv: "61f425f36ad9dded813a958bdeb2fa77"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.142283 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260620 09:07:43.166440 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.168440 13000 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260620 09:07:43.168597 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "db777e9811854a9ca6486e988108f49b"
format_stamp: "Formatted at 2026-06-20 09:07:43 on dist-test-slave-92t4"
server_key: "ea56e62b2ad8af4f0cf24c40779d1076"
server_key_iv: "61f425f36ad9dded813a958bdeb2fa77"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.168759 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:43.211042 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:43.214879 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:43.215077 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:43.215826 13000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:43.216918 13000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:43.216990 13000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.217070 13000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:43.217095 13000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.228057 13000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38405
I20260620 09:07:43.228299 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38405 every 8 connection(s)
I20260620 09:07:43.229300 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:07:43.236208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13000
I20260620 09:07:43.236339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:07:43.236629 392 external_mini_cluster.cc:1428] Setting key c07ccc0100f2856526d8666a5db73a5c
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:43.246676 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:43.231739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53695 (local address 127.0.98.62:40951)
0620 09:07:43.231870 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:43.231874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:43.233086 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 09:07:43.234344 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:43.234348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:43.234410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:43.234526 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:43.235958 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.236793 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.238604 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.238957 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.239034 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:43.242271 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:43.242276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:43.242278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:43.242309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:43.244217 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:43.244840 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:43.244843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:43.244845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:43.244932 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:43.245386 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:43.245389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:43.245391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:43.245608 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:43.245687 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:43.246390 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 09:07:43.246520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:07:43.248286 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:43.232280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.3:53695)
0620 09:07:43.232881 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:43.232942 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:43.234057 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:43.234735 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:43.234744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:43.235082 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:43.235775 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:43.235794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.236959 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.236962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:43.238448 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:43.238461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.239238 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.239242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:43.239456 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:43.240164 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:43.240188 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:43.242054 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:43.244351 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:43.244357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:43.244375 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:43.244720 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:43.245071 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:43.245074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:43.245076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:43.245275 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:43.245750 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:43.245757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:43.246106 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 09:07:43.246900 (+ 794us) client_negotiation.cc:240] Negotiation successful
0620 09:07:43.247257 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":163,"threads_started":1}
I20260620 09:07:43.249926 13129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:43.250178 13129 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:43.250726 13129 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:43.252035 12645 ts_manager.cc:194] Registered new tserver with Master: db777e9811854a9ca6486e988108f49b (127.0.98.3:38405)
I20260620 09:07:43.252743 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53695
I20260620 09:07:43.264271 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:07:43.265123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 12726
I20260620 09:07:43.272526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 12865
I20260620 09:07:43.279225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13000
I20260620 09:07:43.287191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 12613
I20260620 09:07:43.293762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40951
--webserver_interface=127.0.98.62
--webserver_port=39163
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40951
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:43.404232 13133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:43.404512 13133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:43.404560 13133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:43.408561 13133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:43.408625 13133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:43.408644 13133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:43.408660 13133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:43.408675 13133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:43.414275 13133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40951
--ranger_config_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:43.415292 13133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:43.416267 13133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:43.424971 13139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.424989 13138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.424983 13141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.425220 13133 server_base.cc:1034] running on GCE node
I20260620 09:07:43.425871 13133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:43.427073 13133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:43.428272 13133 hybrid_clock.cc:648] HybridClock initialized: now 1781946463428261 us; error 34 us; skew 500 ppm
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:43.431025 13133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:43.431974 13133 webserver.cc:460] Webserver started at http://127.0.98.62:39163/ using document root <none> and password file <none>
I20260620 09:07:43.432529 13133 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:43.432631 13133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:43.479986 13133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:43.482560 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.484262 13133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:07:43.484409 13133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "dc76943db91a41958dcc6655dbc435a4"
format_stamp: "Formatted at 2026-06-20 09:07:41 on dist-test-slave-92t4"
server_key: "20421957c1f56d4882416f2818bde21b"
server_key_iv: "a29037f2bfd4f8dbba17ab00df9e8dd9"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.484997 13133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:43.494503 13133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:43.496919 13133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:43.497144 13133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:43.504334 13133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40951
I20260620 09:07:43.504333 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40951 every 8 connection(s)
I20260620 09:07:43.505550 13133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:43.509972 13202 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:07:43.510847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13133
I20260620 09:07:43.511304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46419
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40951
--webserver_port=43289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:43.511857 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Bootstrap starting.
I20260620 09:07:43.528378 13202 log.cc:824] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:43.534967 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:07:43.535403 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Bootstrap complete.
I20260620 09:07:43.539532 13202 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:43.539959 13202 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc76943db91a41958dcc6655dbc435a4, State: Initialized, Role: FOLLOWER
I20260620 09:07:43.540441 13202 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:43.540598 13202 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:43.540678 13202 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:43.540798 13202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:07:43.542755 13202 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:43.543099 13202 leader_election.cc:302] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc76943db91a41958dcc6655dbc435a4; no voters:
I20260620 09:07:43.543434 13202 leader_election.cc:288] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:07:43.543519 13206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:07:43.543731 13206 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [term 2 LEADER]: Becoming Leader. State: Replica: dc76943db91a41958dcc6655dbc435a4, State: Running, Role: LEADER
I20260620 09:07:43.544021 13206 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } }
I20260620 09:07:43.544685 13202 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:43.545001 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc76943db91a41958dcc6655dbc435a4. Latest consensus state: current_term: 2 leader_uuid: "dc76943db91a41958dcc6655dbc435a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } } }
I20260620 09:07:43.545167 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:43.545377 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc76943db91a41958dcc6655dbc435a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76943db91a41958dcc6655dbc435a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40951 } } }
I20260620 09:07:43.545475 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:43.545646 13210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:43.548904 13210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:43.549422 13202 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 09:07:43.549551 13210 catalog_manager.cc:1259] Loaded cluster ID: 6b1f68e2eef2420eb860a37e794e2575
I20260620 09:07:43.549600 13210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:43.551928 13210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:43.552594 13210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Loaded TSK: 0
I20260620 09:07:43.556174 13210 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:07:43.635473 13204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:43.635905 13204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:43.635982 13204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:43.643172 13204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:43.643260 13204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:43.643352 13204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:07:43.652143 13204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43289
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:43.653618 13204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:43.654953 13204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:43.668159 13239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.668205 13204 server_base.cc:1034] running on GCE node
W20260620 09:07:43.668560 13240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.668282 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.669596 13204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:43.670364 13204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:43.671573 13204 hybrid_clock.cc:648] HybridClock initialized: now 1781946463671551 us; error 39 us; skew 500 ppm
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:43.675268 13204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:07:43.676492 13204 webserver.cc:460] Webserver started at http://127.0.98.1:43289/ using document root <none> and password file <none>
I20260620 09:07:43.677270 13204 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:43.677383 13204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:43.703302 13204 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260620 09:07:43.706852 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.708786 13204 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:07:43.708997 13204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "ff2535abc21bbeb11fd69939e7fa4bf9"
server_key_iv: "94276ae18d9ad8d4f7a661a8ff106bc2"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.709901 13204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:43.738765 13204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:43.742594 13204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:43.742842 13204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:43.743532 13204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:43.744750 13204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:43.744813 13204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.744868 13204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:43.744889 13204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.756111 13204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46419
I20260620 09:07:43.756299 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46419 every 8 connection(s)
I20260620 09:07:43.757468 13204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:07:43.759224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13204
I20260620 09:07:43.759729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42743
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40951
--webserver_port=32971
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:43.782488 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:43.760468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.1:38175)
0620 09:07:43.763345 (+ 2877us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:43.763404 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:43.764697 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:43.766927 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:43.766939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:43.767328 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:43.768059 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:43.768081 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.770302 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.770309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:43.771935 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:43.771950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.773299 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.773305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:43.773490 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:43.774193 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:43.774214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:43.776012 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:43.778610 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:43.778617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:43.778633 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:43.778981 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:43.779404 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:43.779410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:43.779415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:43.779640 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:43.780465 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:43.780471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:43.780808 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 09:07:43.781011 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 09:07:43.781391 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"thread_start_us":147,"threads_started":1}
I20260620 09:07:43.784183 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:43.761799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38175 (local address 127.0.98.62:40951)
0620 09:07:43.762853 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:43.762880 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:43.765081 (+ 2201us) server_negotiation.cc:407] Connection header received
0620 09:07:43.765435 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:43.765454 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:43.765957 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:43.766345 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:43.768278 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.770115 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.772103 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:43.772894 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:43.773070 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:43.776256 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:43.776264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:43.776283 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:43.776316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:43.778453 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:43.779121 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:43.779128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:43.779135 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:43.779282 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:43.779773 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:43.779778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:43.779780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:43.780285 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:43.780402 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:43.782876 (+ 2474us) server_negotiation.cc:299] Negotiation successful
0620 09:07:43.783195 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":118,"threads_started":1}
I20260620 09:07:43.787438 13364 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:43.787700 13364 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:43.788304 13364 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:43.790874 13364 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40951 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:07:43.791029 13364 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:43.791249 13364 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:43.792111 13364 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:43.792303 13364 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:43.793133 13364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:07:43.928682 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:43.929134 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:43.929222 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:43.935503 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:43.935597 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:43.935703 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:07:43.943804 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32971
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:43.945261 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:43.946753 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:43.958896 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:43.958891 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.959188 13367 server_base.cc:1034] running on GCE node
W20260620 09:07:43.959018 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:43.959731 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:43.960554 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:43.961760 13367 hybrid_clock.cc:648] HybridClock initialized: now 1781946463961732 us; error 44 us; skew 500 ppm
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:43.964766 13367 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:07:43.965796 13367 webserver.cc:460] Webserver started at http://127.0.98.2:32971/ using document root <none> and password file <none>
I20260620 09:07:43.966398 13367 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:43.966503 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:07:43 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:43.986837 13367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:07:43.993965 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:43.995741 13367 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260620 09:07:43.995908 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "85bc6ef03df74ce6bce8a71d627a8a77"
format_stamp: "Formatted at 2026-06-20 09:07:42 on dist-test-slave-92t4"
server_key: "2fdc66863751b3dd64d35b825a6c104e"
server_key_iv: "e3a876f9f1534e61a27f9bf7a9494f56"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:43.996528 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:44.019399 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:44.022888 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:44.023106 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:44.023725 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:44.024595 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:44.024667 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:44.024742 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:44.024796 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:44.035110 13367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42743
I20260620 09:07:44.035178 13497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42743 every 8 connection(s)
I20260620 09:07:44.036563 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:07:44.041911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13367
I20260620 09:07:44.042464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51637/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38405
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40951
--webserver_port=41723
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:07:44 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:44.061156 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:44.038751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52983 (local address 127.0.98.62:40951)
0620 09:07:44.038887 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:44.038892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:44.039937 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 09:07:44.040931 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:44.040935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:44.040995 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:44.041144 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:44.042482 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.043229 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.044815 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.045271 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.045411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:44.056504 (+ 11093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:44.056510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:44.056513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:44.056563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:44.058515 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.059243 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.059246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.059248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.059339 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.059745 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.059748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.059750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.059980 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:44.060049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:44.060772 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 09:07:44.060914 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:07:44.062844 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:44.039268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.2:52983)
0620 09:07:44.039740 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:44.039795 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:44.040667 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:44.041357 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:44.041367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:44.041760 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:44.042329 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:44.042343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.043356 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.043360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:44.044666 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:44.044679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.045469 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.045474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:44.045708 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:44.053307 (+ 7599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:44.053333 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:44.056199 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:44.058671 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:44.058679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:44.058701 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:44.059104 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:44.059446 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:44.059449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:44.059451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:44.059639 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:44.060133 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:44.060140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:44.060493 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 09:07:44.061341 (+ 848us) client_negotiation.cc:240] Negotiation successful
0620 09:07:44.061708 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":137,"threads_started":1}
I20260620 09:07:44.064231 13498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:44.064481 13498 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.065086 13498 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:44.066707 13498 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40951 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:07:44.066829 13498 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.067000 13498 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:44.067790 13498 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.067955 13498 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:44.068637 13498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:07:44.195255 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:44.195664 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:44.195739 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:44.202144 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:44.202230 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:44.202323 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:07:44.210459 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51637/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41723
--tserver_master_addrs=127.0.98.62:40951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:44.211916 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:44.213359 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:44.224380 13507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:44.224562 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:44.224380 13508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:44.225510 13501 server_base.cc:1034] running on GCE node
I20260620 09:07:44.225983 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:44.226655 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:44.227826 13501 hybrid_clock.cc:648] HybridClock initialized: now 1781946464227793 us; error 57 us; skew 500 ppm
Jun 20 09:07:44 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:44.235697 13501 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:07:44.237023 13501 webserver.cc:460] Webserver started at http://127.0.98.3:41723/ using document root <none> and password file <none>
I20260620 09:07:44.237862 13501 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:44.238034 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 09:07:44 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:44.261705 13501 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260620 09:07:44.264496 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:44.265985 13501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:44.266144 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "db777e9811854a9ca6486e988108f49b"
format_stamp: "Formatted at 2026-06-20 09:07:43 on dist-test-slave-92t4"
server_key: "ea56e62b2ad8af4f0cf24c40779d1076"
server_key_iv: "61f425f36ad9dded813a958bdeb2fa77"
server_key_version: "kuduclusterkey@0"
I20260620 09:07:44.266737 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:44.295295 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:44.298967 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:44.299193 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:44.299929 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:44.300858 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:44.300906 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:44.300973 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:44.301077 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 09:07:44.313099 13501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38405
I20260620 09:07:44.313118 13631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38405 every 8 connection(s)
I20260620 09:07:44.314383 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:07:44.322286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13501
Jun 20 09:07:44 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:44.333695 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:44.323077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.0.98.62:40951)
0620 09:07:44.323315 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:44.323320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:44.323587 (+ 267us) server_negotiation.cc:407] Connection header received
0620 09:07:44.323759 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:44.323762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:44.323808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:44.323865 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:44.324892 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.325639 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.327636 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.327995 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.328068 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:44.329878 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:44.329884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:44.329887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:44.329939 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:44.331663 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.332127 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.332133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.332138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.332224 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.332556 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.332560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.332562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.332777 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:44.332954 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:44.333341 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 09:07:44.333467 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260620 09:07:44.333842 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:44.318386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43559 (local address 127.0.98.62:40951)
0620 09:07:44.318553 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:44.318559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:44.318581 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:07:44.319015 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:44.319019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:44.319074 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:44.319168 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:44.320426 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.322487 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.324145 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.324543 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.324621 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:44.329658 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:44.329663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:44.329666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:44.329693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:44.331576 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.332207 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.332210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.332212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.332277 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:44.332670 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:44.332673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:44.332674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:44.332825 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:44.332878 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:44.333607 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 09:07:44.333709 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 09:07:44.335817 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:44.317191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40951 (local address 127.0.98.3:43559)
0620 09:07:44.317718 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:44.317760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:44.318749 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:44.319339 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:44.319345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:44.319615 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:44.320261 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:44.320282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.322644 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.322647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:44.324015 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:44.324024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:44.324833 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:44.324837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:44.325065 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:44.325726 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:44.325751 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:44.329451 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:44.331707 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:44.331715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:44.331737 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:44.332108 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:44.332388 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:44.332392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:44.332394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:44.332570 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:44.332955 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:44.332963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:44.333340 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 09:07:44.334353 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0620 09:07:44.334713 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":162,"threads_started":1}
I20260620 09:07:44.336890 13632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40951
I20260620 09:07:44.337185 13632 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.337785 13632 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:44.339401 13632 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40951 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 09:07:44.339570 13632 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.339740 13632 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:44.340524 13632 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.340706 13632 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
W20260620 09:07:44.341423 13632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 09:07:44 dist-test-slave-92t4 krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:44.793375 13364 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:44.793706 13364 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:44.794849 13166 ts_manager.cc:194] Registered new tserver with Master: 893fc5a75a3c4e1ea9c2b617b5aba4e7 (127.0.98.1:46419)
I20260620 09:07:44.796110 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38175
I20260620 09:07:45.068874 13498 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:45.069206 13498 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:45.069968 13166 ts_manager.cc:194] Registered new tserver with Master: 85bc6ef03df74ce6bce8a71d627a8a77 (127.0.98.2:42743)
I20260620 09:07:45.070881 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52983
Jun 20 09:07:45 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:45.341692 13632 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:45.341991 13632 heartbeater.cc:511] Master 127.0.98.62:40951 requested a full tablet report, sending...
I20260620 09:07:45.342834 13166 ts_manager.cc:194] Registered new tserver with Master: db777e9811854a9ca6486e988108f49b (127.0.98.3:38405)
I20260620 09:07:45.343597 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43559
I20260620 09:07:45.350047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 09:07:45 dist-test-slave-92t4 krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 09:07:45.367579 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.354699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.0.98.62:40951)
0620 09:07:45.357013 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.357018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.357054 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:07:45.357107 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.357111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.357165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.358536 (+ 1371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:45.358669 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.359541 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.360829 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.361257 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.363026 (+ 1769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.363160 (+ 134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:45.363164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:45.363167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:45.363184 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:45.365094 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:45.365606 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:45.365609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:45.365611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:45.365696 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:45.366038 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:45.366041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:45.366043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:45.366280 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:45.367178 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.367257 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.367388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":122,"threads_started":1}
I20260620 09:07:45.372342 13166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:07:45.377622 13166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:07:45.418951 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.400790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38405 (local address 127.0.0.1:56026)
0620 09:07:45.411127 (+ 10337us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.411141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:45.411229 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:45.411953 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:45.411968 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:45.411987 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:45.412078 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.412085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.414462 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.414466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.415973 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.415985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.418134 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.418138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.418256 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.418296 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:07:45.418407 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 09:07:45.418768 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10275,"thread_start_us":80,"threads_started":1}
I20260620 09:07:45.421473 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.401078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.98.3:38405)
0620 09:07:45.407859 (+ 6781us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.407868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.411340 (+ 3472us) server_negotiation.cc:407] Connection header received
0620 09:07:45.411429 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.411435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.411646 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.411794 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:45.412218 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.414300 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.416615 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.417623 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.417737 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.417898 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.420706 (+ 2808us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.421329 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6640,"thread_start_us":98,"threads_started":1}
I20260620 09:07:45.422606 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.398640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46419 (local address 127.0.0.1:50716)
0620 09:07:45.409474 (+ 10834us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.409486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:45.409544 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:45.410695 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:45.410703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:45.410730 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:45.410887 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.410895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.417445 (+ 6550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.417451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.420062 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.420072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.422176 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.422180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.422282 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.422317 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:07:45.422394 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:07:45.422468 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9326,"thread_start_us":116,"threads_started":1}
I20260620 09:07:45.423156 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.399378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.0.98.1:46419)
0620 09:07:45.409810 (+ 10432us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.409819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.409846 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:07:45.409934 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.409941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.410300 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.410469 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:45.416136 (+ 5667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.417305 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.420925 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.421822 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.421933 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.422105 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.422892 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.423017 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10315,"thread_start_us":113,"threads_started":1}
I20260620 09:07:45.425613 13550 tablet_service.cc:1467] Processing CreateTablet for tablet 355a952c884649069184bf696161f51b (DEFAULT_TABLE table=test-table [id=ff7cf736d5094def8708aa1ee239a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:45.426400 13279 tablet_service.cc:1467] Processing CreateTablet for tablet 355a952c884649069184bf696161f51b (DEFAULT_TABLE table=test-table [id=ff7cf736d5094def8708aa1ee239a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:45.426896 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355a952c884649069184bf696161f51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:45.427428 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355a952c884649069184bf696161f51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:45.428004 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.400446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42743 (local address 127.0.0.1:45672)
0620 09:07:45.409134 (+ 8688us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.409213 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:45.409352 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:45.418411 (+ 9059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:45.418418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:45.418445 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:45.419437 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.419445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.424456 (+ 5011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.424459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.426468 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.426479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.427623 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.427626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.427717 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.427742 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:07:45.427789 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 09:07:45.427861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8577,"thread_start_us":93,"threads_started":1}
I20260620 09:07:45.428634 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.400622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.0.98.2:42743)
0620 09:07:45.417721 (+ 17099us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.417729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.417756 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:07:45.417835 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.417841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.418099 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.419612 (+ 1513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:45.419752 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.420791 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.426632 (+ 5841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.427476 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.428187 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.428353 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.428405 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.428498 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16965,"thread_start_us":96,"threads_started":1}
I20260620 09:07:45.431342 13430 tablet_service.cc:1467] Processing CreateTablet for tablet 355a952c884649069184bf696161f51b (DEFAULT_TABLE table=test-table [id=ff7cf736d5094def8708aa1ee239a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:45.432355 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355a952c884649069184bf696161f51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:45.445322 13668 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Bootstrap starting.
I20260620 09:07:45.446655 13667 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Bootstrap starting.
I20260620 09:07:45.447947 13668 tablet_bootstrap.cc:654] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:45.448915 13668 log.cc:824] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:45.449015 13667 tablet_bootstrap.cc:654] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:45.449842 13669 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Bootstrap starting.
I20260620 09:07:45.450011 13667 log.cc:824] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:45.452342 13669 tablet_bootstrap.cc:654] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:45.453172 13669 log.cc:824] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:45.455389 13668 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: No bootstrap required, opened a new log
I20260620 09:07:45.455648 13668 ts_tablet_manager.cc:1397] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 09:07:45.460096 13668 raft_consensus.cc:348] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.460528 13668 raft_consensus.cc:374] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:45.460629 13668 raft_consensus.cc:729] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db777e9811854a9ca6486e988108f49b, State: Initialized, Role: FOLLOWER
I20260620 09:07:45.461251 13668 consensus_queue.cc:260] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.462423 13668 ts_tablet_manager.cc:1428] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 09:07:45.462836 13632 heartbeater.cc:503] Master 127.0.98.62:40951 was elected leader, sending a full tablet report...
I20260620 09:07:45.468773 13669 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: No bootstrap required, opened a new log
I20260620 09:07:45.469090 13669 ts_tablet_manager.cc:1397] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260620 09:07:45.476246 13669 raft_consensus.cc:348] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.478889 13667 tablet_bootstrap.cc:492] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: No bootstrap required, opened a new log
I20260620 09:07:45.479120 13667 ts_tablet_manager.cc:1397] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.005s
I20260620 09:07:45.483724 13667 raft_consensus.cc:348] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.484120 13667 raft_consensus.cc:374] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:45.484220 13667 raft_consensus.cc:729] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893fc5a75a3c4e1ea9c2b617b5aba4e7, State: Initialized, Role: FOLLOWER
I20260620 09:07:45.484776 13667 consensus_queue.cc:260] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.486021 13667 ts_tablet_manager.cc:1428] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 09:07:45.486425 13364 heartbeater.cc:503] Master 127.0.98.62:40951 was elected leader, sending a full tablet report...
I20260620 09:07:45.487581 13669 raft_consensus.cc:374] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:45.487703 13669 raft_consensus.cc:729] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bc6ef03df74ce6bce8a71d627a8a77, State: Initialized, Role: FOLLOWER
I20260620 09:07:45.488231 13669 consensus_queue.cc:260] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.489480 13669 ts_tablet_manager.cc:1428] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260620 09:07:45.489909 13498 heartbeater.cc:503] Master 127.0.98.62:40951 was elected leader, sending a full tablet report...
W20260620 09:07:45.509690 13365 tablet.cc:2369] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:07:45.540418 13499 tablet.cc:2369] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:07:45.567230 13633 tablet.cc:2369] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:07:45.665869 13673 raft_consensus.cc:482] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:45.666098 13673 raft_consensus.cc:504] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.667259 13673 leader_election.cc:288] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 893fc5a75a3c4e1ea9c2b617b5aba4e7 (127.0.98.1:46419), 85bc6ef03df74ce6bce8a71d627a8a77 (127.0.98.2:42743)
I20260620 09:07:45.671496 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.667641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52177 (local address 127.0.98.1:46419)
0620 09:07:45.667773 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.667778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.667984 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:07:45.668171 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.668178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.668222 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.668309 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:45.668612 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.669179 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.670465 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.670875 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.671031 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.671113 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.671299 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.671366 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:07:45.671512 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.667516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46419 (local address 127.0.98.3:52177)
0620 09:07:45.667877 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.667889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:45.668033 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:45.668322 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:45.668325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:45.668340 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:45.668493 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.668498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.669294 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.669301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.670342 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.670351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.671058 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.671062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.671155 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.671178 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:07:45.671272 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 09:07:45.671343 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
I20260620 09:07:45.671645 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.667633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39109 (local address 127.0.98.2:42743)
0620 09:07:45.667773 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.667778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.668196 (+ 418us) server_negotiation.cc:407] Connection header received
0620 09:07:45.668343 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.668346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.668401 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.668480 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:45.668761 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.669335 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.670484 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.670950 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.671102 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.671160 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.671452 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.671517 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:07:45.671772 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.667516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42743 (local address 127.0.98.3:39109)
0620 09:07:45.668097 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.668109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:45.668232 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:45.668503 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:45.668506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:45.668520 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:45.668653 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.668659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.669449 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.669456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.670368 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:45.670377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.671249 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.671252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:45.671334 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.671352 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 09:07:45.671435 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 09:07:45.671502 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":323,"thread_start_us":92,"threads_started":1}
I20260620 09:07:45.672228 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355a952c884649069184bf696161f51b" candidate_uuid: "db777e9811854a9ca6486e988108f49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" is_pre_election: true
I20260620 09:07:45.672309 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355a952c884649069184bf696161f51b" candidate_uuid: "db777e9811854a9ca6486e988108f49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" is_pre_election: true
I20260620 09:07:45.672545 13318 raft_consensus.cc:2454] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db777e9811854a9ca6486e988108f49b in term 0.
I20260620 09:07:45.672583 13452 raft_consensus.cc:2454] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db777e9811854a9ca6486e988108f49b in term 0.
I20260620 09:07:45.673004 13519 leader_election.cc:302] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85bc6ef03df74ce6bce8a71d627a8a77, db777e9811854a9ca6486e988108f49b; no voters:
I20260620 09:07:45.673348 13673 raft_consensus.cc:2781] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:07:45.673429 13673 raft_consensus.cc:482] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:07:45.673461 13673 raft_consensus.cc:3037] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:45.674547 13673 raft_consensus.cc:504] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.674950 13673 leader_election.cc:288] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [CANDIDATE]: Term 1 election: Requested vote from peers 893fc5a75a3c4e1ea9c2b617b5aba4e7 (127.0.98.1:46419), 85bc6ef03df74ce6bce8a71d627a8a77 (127.0.98.2:42743)
I20260620 09:07:45.675271 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355a952c884649069184bf696161f51b" candidate_uuid: "db777e9811854a9ca6486e988108f49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7"
I20260620 09:07:45.675326 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355a952c884649069184bf696161f51b" candidate_uuid: "db777e9811854a9ca6486e988108f49b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85bc6ef03df74ce6bce8a71d627a8a77"
I20260620 09:07:45.675405 13318 raft_consensus.cc:3037] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:45.675432 13452 raft_consensus.cc:3037] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:45.676749 13318 raft_consensus.cc:2454] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db777e9811854a9ca6486e988108f49b in term 1.
I20260620 09:07:45.676749 13452 raft_consensus.cc:2454] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db777e9811854a9ca6486e988108f49b in term 1.
I20260620 09:07:45.677019 13519 leader_election.cc:302] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 85bc6ef03df74ce6bce8a71d627a8a77, 893fc5a75a3c4e1ea9c2b617b5aba4e7, db777e9811854a9ca6486e988108f49b; no voters:
I20260620 09:07:45.677175 13673 raft_consensus.cc:2781] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:45.677374 13673 raft_consensus.cc:686] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 1 LEADER]: Becoming Leader. State: Replica: db777e9811854a9ca6486e988108f49b, State: Running, Role: LEADER
I20260620 09:07:45.677702 13673 consensus_queue.cc:237] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } }
I20260620 09:07:45.680733 13166 catalog_manager.cc:5495] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b reported cstate change: term changed from 0 to 1, leader changed from <none> to db777e9811854a9ca6486e988108f49b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "db777e9811854a9ca6486e988108f49b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db777e9811854a9ca6486e988108f49b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 } health_report { overall_health: UNKNOWN } } }
I20260620 09:07:45.693967 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.690196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.0.98.3:38405)
0620 09:07:45.690336 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:45.690341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:45.690355 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:07:45.690537 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:45.690539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:45.690601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:45.690709 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:07:45.690956 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.691457 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.692413 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:45.692639 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:45.692742 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:45.693134 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:07:45.693543 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:07:45.693646 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:45.693718 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 09:07:45.693769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:07:45.699810 13318 raft_consensus.cc:1261] T 355a952c884649069184bf696161f51b P 893fc5a75a3c4e1ea9c2b617b5aba4e7 [term 1 FOLLOWER]: Refusing update from remote peer db777e9811854a9ca6486e988108f49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:45.699810 13452 raft_consensus.cc:1261] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 1 FOLLOWER]: Refusing update from remote peer db777e9811854a9ca6486e988108f49b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:45.700469 13673 consensus_queue.cc:1035] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85bc6ef03df74ce6bce8a71d627a8a77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:45.700703 13685 consensus_queue.cc:1035] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [LEADER]: Connected to new peer: Peer: permanent_uuid: "893fc5a75a3c4e1ea9c2b617b5aba4e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:45.707926 13688 mvcc.cc:205] Tried to move back new op lower bound from 7298852723499008000 to 7298852723418300416. Current Snapshot: MvccSnapshot[applied={T|T < 7298852723499008000}]
I20260620 09:07:45.711761 13690 mvcc.cc:205] Tried to move back new op lower bound from 7298852723499008000 to 7298852723418300416. Current Snapshot: MvccSnapshot[applied={T|T < 7298852723499008000 or (T in {7298852723499008000})}]
I20260620 09:07:45.716964 13166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:45.717187 13166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:45.719504 13166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc76943db91a41958dcc6655dbc435a4: Sending DeleteTablet for 3 replicas of tablet 355a952c884649069184bf696161f51b
I20260620 09:07:45.720355 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet 355a952c884649069184bf696161f51b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45672
I20260620 09:07:45.720407 13550 tablet_service.cc:1514] Processing DeleteTablet for tablet 355a952c884649069184bf696161f51b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56026
I20260620 09:07:45.720477 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet 355a952c884649069184bf696161f51b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50716
I20260620 09:07:45.720928 13698 tablet_replica.cc:330] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: stopping tablet replica
I20260620 09:07:45.721091 13697 tablet_replica.cc:330] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: stopping tablet replica
I20260620 09:07:45.721343 13698 raft_consensus.cc:2229] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:07:45.721412 13697 raft_consensus.cc:2229] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:07:45.721539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13204
I20260620 09:07:45.722302 13698 raft_consensus.cc:2258] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:45.722306 13697 raft_consensus.cc:2258] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:45.724043 13698 ts_tablet_manager.cc:1905] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:45.724058 13697 ts_tablet_manager.cc:1905] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:45.726876 13698 ts_tablet_manager.cc:1918] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:45.727016 13698 log.cc:1196] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/355a952c884649069184bf696161f51b
I20260620 09:07:45.727416 13698 ts_tablet_manager.cc:1939] T 355a952c884649069184bf696161f51b P 85bc6ef03df74ce6bce8a71d627a8a77: Deleting consensus metadata
I20260620 09:07:45.727497 13697 ts_tablet_manager.cc:1918] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:45.727670 13697 log.cc:1196] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/355a952c884649069184bf696161f51b
I20260620 09:07:45.728045 13697 ts_tablet_manager.cc:1939] T 355a952c884649069184bf696161f51b P db777e9811854a9ca6486e988108f49b: Deleting consensus metadata
I20260620 09:07:45.728750 13150 catalog_manager.cc:4841] TS 85bc6ef03df74ce6bce8a71d627a8a77 (127.0.98.2:42743): tablet 355a952c884649069184bf696161f51b (table test-table [id=ff7cf736d5094def8708aa1ee239a2b0]) successfully deleted
I20260620 09:07:45.729450 13153 catalog_manager.cc:4841] TS db777e9811854a9ca6486e988108f49b (127.0.98.3:38405): tablet 355a952c884649069184bf696161f51b (table test-table [id=ff7cf736d5094def8708aa1ee239a2b0]) successfully deleted
W20260620 09:07:45.729957 13153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46419 (error 108)
I20260620 09:07:45.730213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13367
I20260620 09:07:45.731804 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:45.731333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46419 (local address 127.0.0.1:50724)
0620 09:07:45.731673 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:45.731734 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
W20260620 09:07:45.731951 13153 catalog_manager.cc:4584] TS 893fc5a75a3c4e1ea9c2b617b5aba4e7 (127.0.98.1:46419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 355a952c884649069184bf696161f51b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46419: connect: Connection refused (error 111)
I20260620 09:07:45.736814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13501
I20260620 09:07:45.741779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13133
I20260620 09:07:45.747830 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 09:07:46.082839 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 09:07:46.082974 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 09:07:46.428215 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 09:07:46.428 UTC [12199] LOG: received smart shutdown request
2026-06-20 09:07:46.432 UTC [12199] LOG: background worker "logical replication launcher" (PID 12208) exited with exit code 1
2026-06-20 09:07:46.432 UTC [12203] LOG: shutting down
2026-06-20 09:07:46.592 UTC [12199] LOG: database system is shut down
2026-06-20T09:07:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47054 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:07:46 dist-test-slave-92t4 krb5kdc[13709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:07:46 dist-test-slave-92t4 krb5kdc[13709](info): set up 2 sockets
Jun 20 09:07:46 dist-test-slave-92t4 krb5kdc[13709](info): commencing operation
krb5kdc: starting...
W20260620 09:07:47.765378 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:07:47 dist-test-slave-92t4 krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:07:47.888832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:48.000061 13722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:48.000399 13722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:48.000479 13722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:48.004693 13722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:48.004787 13722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:48.004824 13722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:48.004860 13722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:48.004895 13722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:48.010435 13722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:48.011579 13722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:48.012650 13722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:48.021381 13728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:48.021487 13722 server_base.cc:1034] running on GCE node
W20260620 09:07:48.021364 13730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:48.021395 13727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:48.022293 13722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:48.023386 13722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:48.024642 13722 hybrid_clock.cc:648] HybridClock initialized: now 1781946468024613 us; error 43 us; skew 500 ppm
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:48.027670 13722 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:48.028592 13722 webserver.cc:460] Webserver started at http://127.0.98.62:45351/ using document root <none> and password file <none>
I20260620 09:07:48.029268 13722 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:48.029361 13722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:48.029597 13722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:48.031700 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "9f3323748bfd4cfba9682bb5628df9ed"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "2446bfc30200e0817f8ef2e42e9a5ceb"
server_key_iv: "82196e683f88406e236cbeb051768367"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.032295 13722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f3323748bfd4cfba9682bb5628df9ed"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "2446bfc30200e0817f8ef2e42e9a5ceb"
server_key_iv: "82196e683f88406e236cbeb051768367"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.036432 13722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 09:07:48.039018 13737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:48.040274 13722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:07:48.040434 13722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "9f3323748bfd4cfba9682bb5628df9ed"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "2446bfc30200e0817f8ef2e42e9a5ceb"
server_key_iv: "82196e683f88406e236cbeb051768367"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.040565 13722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:48.066936 13722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:48.070436 13722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:48.070616 13722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:48.078437 13722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46347
I20260620 09:07:48.078512 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46347 every 8 connection(s)
I20260620 09:07:48.079530 13722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:48.082554 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:48.085354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13722
I20260620 09:07:48.085498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:07:48.085796 392 external_mini_cluster.cc:1428] Setting key 0e6c95e9282acaab55a4d8ce04b076c1
I20260620 09:07:48.086609 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:48.103127 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.089370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46347 (local address 127.0.0.1:45974)
0620 09:07:48.089985 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.090043 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.090868 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.091701 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.091705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.092020 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.092400 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.092413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.094176 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.094178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.094896 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.094904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.095520 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.095523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.095612 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.096221 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.096237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.098134 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.100043 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.100050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.100060 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.100330 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.100640 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.100645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.100647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.100761 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.101420 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.101425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.101571 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.101739 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.102130 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":152,"spinlock_wait_cycles":768,"thread_start_us":165,"threads_started":1}
I20260620 09:07:48.103142 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.089370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.98.62:46347)
0620 09:07:48.089880 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.089901 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.090164 (+ 263us) server_negotiation.cc:407] Connection header received
0620 09:07:48.091200 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.091209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.091484 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.091700 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.092557 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.094053 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.095013 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.095383 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.095513 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.098294 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.098300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.098310 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.098330 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.099914 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.100438 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.100449 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.100451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.100548 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.100858 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.100862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.100863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.101302 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.101439 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.101809 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.102081 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260620 09:07:48.105548 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:07:48.106386 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.106129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32946)
0620 09:07:48.106258 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.106326 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 09:07:48.106623 13739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
I20260620 09:07:48.107537 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.107320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32948)
0620 09:07:48.107432 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.107486 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:07:48.107689 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:07:48.112671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45211
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:48.156981 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } attempt: 1
I20260620 09:07:48.157940 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.157639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32950)
0620 09:07:48.157791 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.157864 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 09:07:48.158782 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.158552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32956)
0620 09:07:48.158679 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.158725 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 09:07:48.159022 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
I20260620 09:07:48.227233 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } attempt: 2
I20260620 09:07:48.228067 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.227759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32966)
0620 09:07:48.227916 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.227988 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 09:07:48.228961 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.228705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32970)
0620 09:07:48.228835 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.228889 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 09:07:48.229136 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45211: connect: Connection refused (error 111)
W20260620 09:07:48.231091 13797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:48.231356 13797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:48.231439 13797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:48.235634 13797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:48.235692 13797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:48.235713 13797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:48.235729 13797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:48.235744 13797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:48.241407 13797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:48.242558 13797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:48.243674 13797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:48.252763 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:48.252770 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:48.252787 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:48.253057 13797 server_base.cc:1034] running on GCE node
I20260620 09:07:48.253695 13797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:48.254781 13797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:48.255934 13797 hybrid_clock.cc:648] HybridClock initialized: now 1781946468255898 us; error 52 us; skew 500 ppm
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:48.258843 13797 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:07:48.259796 13797 webserver.cc:460] Webserver started at http://127.0.98.61:35515/ using document root <none> and password file <none>
I20260620 09:07:48.260365 13797 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:48.260455 13797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:48.260699 13797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:48.262781 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data/instance:
uuid: "d2730c6068874961b01c526ed0a436d7"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "b9ed5b5b0ec1bbb2e9395d111f4e1b7c"
server_key_iv: "0898ef389584187fe23dd9ceb1fb5b21"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.263351 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal/instance:
uuid: "d2730c6068874961b01c526ed0a436d7"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "b9ed5b5b0ec1bbb2e9395d111f4e1b7c"
server_key_iv: "0898ef389584187fe23dd9ceb1fb5b21"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.267149 13797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:07:48.269548 13812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:48.270720 13797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:07:48.270859 13797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "d2730c6068874961b01c526ed0a436d7"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "b9ed5b5b0ec1bbb2e9395d111f4e1b7c"
server_key_iv: "0898ef389584187fe23dd9ceb1fb5b21"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.270972 13797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:48.285534 13797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:48.289394 13797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:48.289613 13797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:48.296691 13797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45211
I20260620 09:07:48.296716 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45211 every 8 connection(s)
I20260620 09:07:48.297752 13797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:07:48.300444 13865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:48.300992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13797
I20260620 09:07:48.301162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-1/wal/instance
I20260620 09:07:48.301448 392 external_mini_cluster.cc:1428] Setting key 93c7717124eb9198c313773b35643156
I20260620 09:07:48.304250 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:48.318737 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.306262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.62:46347)
0620 09:07:48.306429 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.306434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.307389 (+ 955us) server_negotiation.cc:407] Connection header received
0620 09:07:48.308678 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.308681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.308728 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.308800 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.310223 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.310851 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.311942 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.312229 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.312311 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.315100 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.315103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.315105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.315130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.316625 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.317169 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.317173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.317174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.317258 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.317611 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.317614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.317616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.317802 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.317909 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.318429 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.318558 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:07:48.319533 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.306650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46347 (local address 127.0.0.1:45988)
0620 09:07:48.307184 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.307250 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.308431 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.309014 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.309024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.309545 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.310070 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.310083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.310963 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.310965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.311826 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.311835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.312382 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.312385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.312526 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.313179 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.313195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.314926 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.316746 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.316752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.316767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.317078 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.317353 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.317356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.317358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.317508 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.317905 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.317911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.318169 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.318354 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.318691 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260620 09:07:48.320287 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } attempt: 3
I20260620 09:07:48.321162 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:07:48.330279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44029
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:48.331310 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.320728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32974)
0620 09:07:48.320852 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.320872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.320961 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.321611 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.321614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.321784 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.322386 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.322399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.323522 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.323525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.324256 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.324265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.324774 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.324901 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.325468 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.325487 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.326734 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.329303 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.329309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.329315 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.329574 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.329883 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.329888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.329890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.329992 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.330643 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.330649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.330772 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.330893 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.331050 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 09:07:48.331329 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.320834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.0.98.61:45211)
0620 09:07:48.321227 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.321240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.321264 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:07:48.321303 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.321308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.321501 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.321635 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.322523 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.323413 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324370 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.324652 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324763 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.326878 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.326887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.326901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.327000 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.329180 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.329698 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.329704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.329706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.329790 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.330076 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.330082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.330087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.330445 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.330579 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.330994 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.331138 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":298,"spinlock_wait_cycles":31488,"thread_start_us":120,"threads_started":1}
I20260620 09:07:48.331715 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.321811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.0.98.61:45211)
0620 09:07:48.322067 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.322073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.322083 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:07:48.322114 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.322116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.322168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.322274 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.322682 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.323421 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324159 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.324672 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324774 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.327386 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.327388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.327390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.327409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.329889 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.330351 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.330356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.330358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.330438 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.330700 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.330705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.330707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.330885 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.330982 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.331467 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.331568 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260620 09:07:48.331888 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.321676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32982)
0620 09:07:48.321774 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.321787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.321850 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.322282 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.322285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.322449 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.322571 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.322578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.323547 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.323551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.324052 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.324059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.324772 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.324775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.324994 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.325596 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.325607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.327222 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.329989 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.329992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.329994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.330256 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.330534 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.330537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.330539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.330615 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.331016 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.331018 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.331087 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.331185 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.331286 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":113,"spinlock_wait_cycles":1408}
I20260620 09:07:48.333273 13790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:07:48.333902 13865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:07:48.334059 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.333729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35408)
0620 09:07:48.333929 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.334005 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 09:07:48.334601 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.334393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35418)
0620 09:07:48.334478 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.334549 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 09:07:48.334828 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.334488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35424)
0620 09:07:48.334685 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.334745 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 09:07:48.334812 13815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
W20260620 09:07:48.335026 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
I20260620 09:07:48.335729 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.335523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35434)
0620 09:07:48.335627 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.335669 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:07:48.335872 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
I20260620 09:07:48.366226 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 1
I20260620 09:07:48.367219 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.366874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35440)
0620 09:07:48.367064 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.367135 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 09:07:48.368008 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.367777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35454)
0620 09:07:48.367874 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.367953 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:07:48.368171 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
I20260620 09:07:48.385156 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 1
I20260620 09:07:48.386052 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.385726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35466)
0620 09:07:48.385893 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.385961 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 09:07:48.386958 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.386622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35472)
0620 09:07:48.386796 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.386858 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 09:07:48.387130 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
I20260620 09:07:48.443378 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 2
I20260620 09:07:48.444342 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.443990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35474)
0620 09:07:48.444151 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.444218 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 09:07:48.445179 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.444884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35490)
0620 09:07:48.445030 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.445113 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 09:07:48.445313 13790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
W20260620 09:07:48.445998 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:48.446300 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:48.446393 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:48.450527 13872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:48.450615 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:48.450660 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:48.450695 13872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:48.450726 13872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:48.455309 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 2
I20260620 09:07:48.456075 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.455796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35498)
0620 09:07:48.455931 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.455997 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 09:07:48.456351 13872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46347,127.0.98.61:45211,127.0.98.60:44029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:48.456817 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.456604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35502)
0620 09:07:48.456711 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.456762 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:07:48.456988 13865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44029: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44029: connect: Connection refused (error 111)
I20260620 09:07:48.457727 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:48.458771 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:48.467520 13878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:48.467557 13872 server_base.cc:1034] running on GCE node
W20260620 09:07:48.467543 13881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:48.467520 13879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:48.468284 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:48.469446 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:48.470606 13872 hybrid_clock.cc:648] HybridClock initialized: now 1781946468470564 us; error 54 us; skew 500 ppm
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:48.473338 13872 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:07:48.474323 13872 webserver.cc:460] Webserver started at http://127.0.98.60:33377/ using document root <none> and password file <none>
I20260620 09:07:48.474861 13872 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:48.474943 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:48.475144 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:48.477392 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data/instance:
uuid: "cbfaf64e576a45f98c7d60c2d68e73f1"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "cd8dd8458debf2eeb275f1de9001725b"
server_key_iv: "8cba58dcb377c7ae010e7570aff08258"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.477931 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal/instance:
uuid: "cbfaf64e576a45f98c7d60c2d68e73f1"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "cd8dd8458debf2eeb275f1de9001725b"
server_key_iv: "8cba58dcb377c7ae010e7570aff08258"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.481737 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:07:48.484308 13888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:48.485703 13872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:48.485841 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "cbfaf64e576a45f98c7d60c2d68e73f1"
format_stamp: "Formatted at 2026-06-20 09:07:48 on dist-test-slave-92t4"
server_key: "cd8dd8458debf2eeb275f1de9001725b"
server_key_iv: "8cba58dcb377c7ae010e7570aff08258"
server_key_version: "encryptionkey@0"
I20260620 09:07:48.485976 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:48.502840 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:48.505965 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:48.506171 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:48.513566 13872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44029
I20260620 09:07:48.513590 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44029 every 8 connection(s)
I20260620 09:07:48.514561 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:07:48.517373 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:48.517611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 13872
I20260620 09:07:48.517755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781946309842113-392-0/minicluster-data/master-2/wal/instance
I20260620 09:07:48.518038 392 external_mini_cluster.cc:1428] Setting key e7a7f26fa7c1d8c4985fdbf4ba2b5871
I20260620 09:07:48.522476 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:48.533298 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.519453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.0.98.60:44029)
0620 09:07:48.519892 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.519911 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.519954 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:07:48.520736 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.520762 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.521218 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.521635 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.522318 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.523761 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.524889 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.525601 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.525760 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.528430 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.528437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.528452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.528480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.530484 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.530917 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.530922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.530928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.531039 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.531336 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.531340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.531342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.531722 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.531913 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.532216 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.532493 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260620 09:07:48.533913 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.524485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.0.98.62:46347)
0620 09:07:48.524632 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.524638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.524979 (+ 341us) server_negotiation.cc:407] Connection header received
0620 09:07:48.525172 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.525175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.525232 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.525308 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.526133 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.526715 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.527868 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.528140 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.528249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.530619 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.530622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.530624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.530647 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.532069 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.532591 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.532599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.532600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.532668 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.532937 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.532942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.532944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.533185 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.533288 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.533709 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.533806 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:07:48.533951 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.524418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46347 (local address 127.0.0.1:45992)
0620 09:07:48.524818 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.524885 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.525073 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.525380 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.525385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.525834 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.526005 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.526011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.526851 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.526855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.527746 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.527755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.528285 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.528289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.528394 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.528881 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.528892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.530477 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.532180 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.532190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.532195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.532494 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.532759 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.532763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.532765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.532850 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.533287 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.533293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.533415 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.533614 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.533774 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":179,"threads_started":1}
I20260620 09:07:48.535171 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:48.543963 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.535644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45211 (local address 127.0.0.1:32984)
0620 09:07:48.535785 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.535799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.535864 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.536111 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.536113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.536283 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.536385 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.536391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.537360 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.537363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.537933 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.537941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.538475 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.538481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.538579 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.539040 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.539056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.540465 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.542362 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.542365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.542367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.542612 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.542873 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.542879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.542882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.542974 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.543426 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.543432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.543538 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.543660 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.543793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 09:07:48.544060 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.535761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.0.98.61:45211)
0620 09:07:48.535887 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.535892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.535909 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:07:48.535957 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.535960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.536010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.536091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.536515 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.537262 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.538048 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.538356 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.538436 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.540609 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.540612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.540614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.540638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.542254 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.542707 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.542710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.542711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.542786 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.543071 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.543074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.543076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.543333 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.543407 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.543717 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.543892 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:07:48.544480 13790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 3
I20260620 09:07:48.545100 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:07:48.545171 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.535675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.62:46347)
0620 09:07:48.535779 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.535782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.535976 (+ 194us) server_negotiation.cc:407] Connection header received
0620 09:07:48.536093 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.536096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.536139 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.536222 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.536881 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.537426 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.538201 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.538392 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.538505 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.541162 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.541166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.541169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.541199 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.542839 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.543283 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.543285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.543286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.543353 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.543652 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.543655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.543657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.543829 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.544066 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.544269 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.544380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:07:48.548182 13865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } attempt: 3
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:07:48 dist-test-slave-92t4 krb5kdc[13709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:07:48.555269 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.544889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35512)
0620 09:07:48.544994 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.545013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.545124 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.545347 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.545350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.545519 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.545607 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.545615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.547411 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.547415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.548236 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.548246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.548841 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.548845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.548928 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.549365 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.549378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.551484 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.553512 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.553517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.553521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.553785 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.554131 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.554138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.554143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.554249 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.554717 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.554720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.554823 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.554912 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.555071 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 09:07:48.556157 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.546532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.0.98.61:45211)
0620 09:07:48.546642 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.546646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.546658 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:07:48.546692 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.546695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.546739 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.546816 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.547512 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.548243 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.549306 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.549644 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.549756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.552434 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.552437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.552440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.552463 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.554219 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.554711 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.554715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.554717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.554795 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.555129 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.555133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.555136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.555340 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.555517 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.555895 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.556006 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 09:07:48.556285 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.544958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.98.60:44029)
0620 09:07:48.545093 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.545097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.545110 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:07:48.545206 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.545209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.545254 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.545333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.545737 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.547279 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.548380 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.548713 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.549165 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.551613 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.551618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.551621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.551644 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.553382 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.553914 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.553919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.553922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.554020 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.554377 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.554382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.554385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.554598 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.555799 (+ 1201us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.555878 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.556017 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260620 09:07:48.556818 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.545544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35518)
0620 09:07:48.545624 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.545636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.545905 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.546176 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.546182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.546345 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.546439 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.546461 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.547380 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.547386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.547875 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.547881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.548550 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.548553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.548628 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.549368 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.549378 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.552168 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.555032 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.555037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.555039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.555244 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.555532 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.555536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.555539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.555867 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.556333 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.556336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.556429 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.556545 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.556664 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 09:07:48.557754 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.545663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.0.98.60:44029)
0620 09:07:48.545982 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.545990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.546002 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:07:48.546032 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.546034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.546069 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.546157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.546572 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.547275 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.548001 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.548331 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.548429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.552306 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.552311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.552314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.552341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.554935 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.555342 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.555346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.555348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.555423 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.555959 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.555963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.555967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.556212 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.556296 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.557028 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.557153 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260620 09:07:48.559613 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.548576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44029 (local address 127.0.0.1:35524)
0620 09:07:48.548805 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:48.548821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:48.548887 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:48.550210 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:48.550215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:48.550411 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:48.550519 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.550528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.551707 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.551711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.552514 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:48.552522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.553223 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.553227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:48.553299 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.553716 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:48.553730 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:48.555210 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:48.558161 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.558164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.558165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.558391 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.558658 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:48.558661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:48.558663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:48.558748 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:48.559125 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:48.559130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:48.559219 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 09:07:48.559325 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:07:48.559459 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 09:07:48.559641 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.549082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.0.98.60:44029)
0620 09:07:48.550013 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.550019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.550029 (+ 10us) server_negotiation.cc:407] Connection header received
0620 09:07:48.550055 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.550057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.550087 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.550156 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.550657 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.551538 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.552666 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.553086 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.553512 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.555344 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.555348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.555350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.555376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.558020 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.558489 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.558491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.558492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.558553 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.558848 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.558851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.558853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.559012 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.559111 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.559374 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.559474 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":92,"threads_started":1}
I20260620 09:07:48.559937 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: Bootstrap starting.
I20260620 09:07:48.560586 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: Bootstrap starting.
I20260620 09:07:48.563194 13941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:48.563604 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:07:48.563869 13865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: Bootstrap starting.
I20260620 09:07:48.564141 13941 log.cc:824] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:48.566459 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: No bootstrap required, opened a new log
I20260620 09:07:48.566654 13865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:48.567472 13865 log.cc:824] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:48.569003 13790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:48.569875 13790 log.cc:824] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:48.572141 13865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: No bootstrap required, opened a new log
I20260620 09:07:48.572342 13941 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.572597 13941 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:48.572651 13941 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfaf64e576a45f98c7d60c2d68e73f1, State: Initialized, Role: FOLLOWER
I20260620 09:07:48.573148 13941 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.575341 13941 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:48.575649 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.576382 13865 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.576589 13865 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:48.576638 13865 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2730c6068874961b01c526ed0a436d7, State: Initialized, Role: FOLLOWER
I20260620 09:07:48.577117 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.565923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.98.61:45211)
0620 09:07:48.566032 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.566037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.566584 (+ 547us) server_negotiation.cc:407] Connection header received
0620 09:07:48.566773 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.566776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.566821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.566871 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.567944 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.568726 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.570127 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.570461 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.570534 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.572934 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.572941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.572947 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.572975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.574808 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.575542 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.575545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.575547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.575657 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.576078 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.576081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.576082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.576293 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.576397 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.576775 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.576892 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:07:48.577438 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.565924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.62:46347)
0620 09:07:48.566032 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.566037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.566614 (+ 577us) server_negotiation.cc:407] Connection header received
0620 09:07:48.566774 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.566777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.566821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.566871 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.567944 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.568733 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.570121 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.570461 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.570534 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.572991 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.572994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.572997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.573026 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.574947 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.575541 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.575545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.575547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.575776 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.576075 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.576079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.576081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.576293 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.576405 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.577161 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.577283 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:07:48.575798 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.578007 13865 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.579638 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: No bootstrap required, opened a new log
I20260620 09:07:48.581346 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.581493 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.579005 13872 master_runner.cc:186] Error getting master registration for 127.0.98.62:46347: OK, instance_id { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" instance_seqno: 1781946468076152 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:07:48.582198 13872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:07:48.582231 13865 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:07:48.587021 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:07:48.587181 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:07:48.587153 13790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.587464 13790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:48.587563 13790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3323748bfd4cfba9682bb5628df9ed, State: Initialized, Role: FOLLOWER
I20260620 09:07:48.588112 13790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
W20260620 09:07:48.588344 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:07:48.588418 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:07:48.585429 13797 master_runner.cc:186] Error getting master registration for 127.0.98.62:46347: OK, instance_id { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" instance_seqno: 1781946468076152 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:07:48.589151 13797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:07:48.589712 13790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: configured and running, proceeding with master startup.
W20260620 09:07:48.594731 13994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:07:48.594863 13994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:07:48.597682 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.597883 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.601809 13722 master_runner.cc:263] No leader master found from master 9f3323748bfd4cfba9682bb5628df9ed
I20260620 09:07:48.601874 13722 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:07:48.602242 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:48.585475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.0.98.60:44029)
0620 09:07:48.585592 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:48.585596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:48.588897 (+ 3301us) server_negotiation.cc:407] Connection header received
0620 09:07:48.589023 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:48.589026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:48.589104 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:48.589187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:48.590221 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.591050 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.592246 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:48.592599 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:48.592696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:48.594533 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:48.594537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:48.594540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:48.594568 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:48.596246 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.596747 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.596752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.596754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.596840 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:48.597177 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:48.597180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:48.597182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:48.597403 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:48.597567 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:48.598215 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 09:07:48.602076 (+ 3861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:07:48.728621 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:48.728853 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.730072 13969 leader_election.cc:288] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f3323748bfd4cfba9682bb5628df9ed (127.0.98.62:46347), cbfaf64e576a45f98c7d60c2d68e73f1 (127.0.98.60:44029)
I20260620 09:07:48.731109 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2730c6068874961b01c526ed0a436d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" is_pre_election: true
I20260620 09:07:48.731503 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2730c6068874961b01c526ed0a436d7 in term 0.
I20260620 09:07:48.732159 13815 leader_election.cc:302] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbfaf64e576a45f98c7d60c2d68e73f1, d2730c6068874961b01c526ed0a436d7; no voters:
I20260620 09:07:48.732580 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:07:48.732669 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:07:48.732710 13969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:48.734189 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.734907 13764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2730c6068874961b01c526ed0a436d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3323748bfd4cfba9682bb5628df9ed" is_pre_election: true
I20260620 09:07:48.735214 13764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2730c6068874961b01c526ed0a436d7 in term 0.
I20260620 09:07:48.736107 13969 leader_election.cc:288] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [CANDIDATE]: Term 1 election: Requested vote from peers 9f3323748bfd4cfba9682bb5628df9ed (127.0.98.62:46347), cbfaf64e576a45f98c7d60c2d68e73f1 (127.0.98.60:44029)
I20260620 09:07:48.736532 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2730c6068874961b01c526ed0a436d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1"
I20260620 09:07:48.736660 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:48.737969 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2730c6068874961b01c526ed0a436d7 in term 1.
I20260620 09:07:48.738760 13764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2730c6068874961b01c526ed0a436d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f3323748bfd4cfba9682bb5628df9ed"
I20260620 09:07:48.738899 13764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:48.740242 13764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2730c6068874961b01c526ed0a436d7 in term 1.
I20260620 09:07:48.738370 13815 leader_election.cc:302] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbfaf64e576a45f98c7d60c2d68e73f1, d2730c6068874961b01c526ed0a436d7; no voters:
I20260620 09:07:48.741008 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:48.741351 13969 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [term 1 LEADER]: Becoming Leader. State: Replica: d2730c6068874961b01c526ed0a436d7, State: Running, Role: LEADER
I20260620 09:07:48.741740 13969 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } }
I20260620 09:07:48.743588 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2730c6068874961b01c526ed0a436d7. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.746610 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:48.748466 13998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:48.749537 13998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:48.755426 13915 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [term 1 FOLLOWER]: Refusing update from remote peer d2730c6068874961b01c526ed0a436d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:48.756233 13996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:07:48.757556 13894 tablet.cc:2369] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:07:48.760735 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2730c6068874961b01c526ed0a436d7. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.760862 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.761484 13764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [term 1 FOLLOWER]: Refusing update from remote peer d2730c6068874961b01c526ed0a436d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:48.762249 13996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:48.767234 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2730c6068874961b01c526ed0a436d7. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.767382 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.768698 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.768824 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:48.769708 13999 mvcc.cc:205] Tried to move back new op lower bound from 7298852736014168064 to 7298852735969579008. Current Snapshot: MvccSnapshot[applied={T|T < 7298852736014168064}]
I20260620 09:07:48.770393 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.770524 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.771844 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.771961 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:48.772846 14002 mvcc.cc:205] Tried to move back new op lower bound from 7298852736014168064 to 7298852735969579008. Current Snapshot: MvccSnapshot[applied={T|T < 7298852736014168064}]
I20260620 09:07:48.773118 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2730c6068874961b01c526ed0a436d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3323748bfd4cfba9682bb5628df9ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46347 } } peers { permanent_uuid: "d2730c6068874961b01c526ed0a436d7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45211 } } peers { permanent_uuid: "cbfaf64e576a45f98c7d60c2d68e73f1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44029 } } }
I20260620 09:07:48.773232 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:07:48.774147 14000 mvcc.cc:205] Tried to move back new op lower bound from 7298852736014168064 to 7298852735969579008. Current Snapshot: MvccSnapshot[applied={T|T < 7298852736014168064}]
I20260620 09:07:48.775678 13998 catalog_manager.cc:1347] Generated new cluster ID: c76f6fe762cb417b882725a76e33f4f8
I20260620 09:07:48.775745 13998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:48.788038 13998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:07:48.789400 13998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:48.798727 13998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2730c6068874961b01c526ed0a436d7: Generated new TSK 0
I20260620 09:07:48.799813 13998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:07:49.590046 13968 catalog_manager.cc:1259] Loaded cluster ID: c76f6fe762cb417b882725a76e33f4f8
I20260620 09:07:49.590130 13968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: loading cluster ID for follower catalog manager: success
I20260620 09:07:49.592083 13968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: acquiring CA information for follower catalog manager: success
I20260620 09:07:49.593290 13968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbfaf64e576a45f98c7d60c2d68e73f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:07:49.596745 13994 catalog_manager.cc:1259] Loaded cluster ID: c76f6fe762cb417b882725a76e33f4f8
I20260620 09:07:49.596834 13994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: loading cluster ID for follower catalog manager: success
I20260620 09:07:49.598476 13994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: acquiring CA information for follower catalog manager: success
I20260620 09:07:49.599611 13994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f3323748bfd4cfba9682bb5628df9ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:07:49.867316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13722
I20260620 09:07:49.873878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13797
I20260620 09:07:49.880079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 13872
2026-06-20T09:07:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3166 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:07:49 dist-test-slave-92t4 krb5kdc[14012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:07:49 dist-test-slave-92t4 krb5kdc[14012](info): set up 2 sockets
Jun 20 09:07:49 dist-test-slave-92t4 krb5kdc[14012](info): commencing operation
krb5kdc: starting...
W20260620 09:07:51.005350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:07:51.131850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:51.244916 14025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:51.245289 14025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:51.245393 14025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:51.249800 14025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:51.249881 14025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:51.249900 14025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:51.249918 14025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:51.250012 14025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:51.255748 14025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:51.257009 14025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:51.258236 14025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:51.267323 14031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:51.267341 14033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:51.267351 14030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:51.267633 14025 server_base.cc:1034] running on GCE node
I20260620 09:07:51.268268 14025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:51.269487 14025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:51.270690 14025 hybrid_clock.cc:648] HybridClock initialized: now 1781946471270675 us; error 30 us; skew 500 ppm
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:51.273801 14025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:51.274776 14025 webserver.cc:460] Webserver started at http://127.0.98.62:35591/ using document root <none> and password file <none>
I20260620 09:07:51.275390 14025 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:51.275484 14025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:51.275728 14025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:51.278050 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "3e45e00135db45e9a6ef9f91078269ca"
format_stamp: "Formatted at 2026-06-20 09:07:51 on dist-test-slave-92t4"
server_key: "dcfc1294fe106941b4763749785525c4"
server_key_iv: "ce6a33c6c55f9276d672bc636ca86f9d"
server_key_version: "encryptionkey@0"
I20260620 09:07:51.278707 14025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e45e00135db45e9a6ef9f91078269ca"
format_stamp: "Formatted at 2026-06-20 09:07:51 on dist-test-slave-92t4"
server_key: "dcfc1294fe106941b4763749785525c4"
server_key_iv: "ce6a33c6c55f9276d672bc636ca86f9d"
server_key_version: "encryptionkey@0"
I20260620 09:07:51.283051 14025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:07:51.285919 14040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:51.287375 14025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:07:51.287544 14025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "3e45e00135db45e9a6ef9f91078269ca"
format_stamp: "Formatted at 2026-06-20 09:07:51 on dist-test-slave-92t4"
server_key: "dcfc1294fe106941b4763749785525c4"
server_key_iv: "ce6a33c6c55f9276d672bc636ca86f9d"
server_key_version: "encryptionkey@0"
I20260620 09:07:51.287679 14025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:51.308023 14025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:51.311476 14025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:51.311715 14025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:51.319720 14025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33205
I20260620 09:07:51.319748 14092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33205 every 8 connection(s)
I20260620 09:07:51.320796 14025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:51.323720 14093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:51.328826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14025
I20260620 09:07:51.328939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:07:51.329277 392 external_mini_cluster.cc:1428] Setting key f6d638bed43a436b9e5c1d63527f0fee
I20260620 09:07:51.329934 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca: Bootstrap starting.
I20260620 09:07:51.332644 14093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:51.333563 14093 log.cc:824] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:51.335896 14093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca: No bootstrap required, opened a new log
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:51.340770 14093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } }
I20260620 09:07:51.341152 14093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:51.341230 14093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e45e00135db45e9a6ef9f91078269ca, State: Initialized, Role: FOLLOWER
I20260620 09:07:51.341854 14093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } }
I20260620 09:07:51.342028 14093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:51.342096 14093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:51.342195 14093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:51.343475 14093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } }
I20260620 09:07:51.343880 14093 leader_election.cc:302] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e45e00135db45e9a6ef9f91078269ca; no voters:
I20260620 09:07:51.344234 14093 leader_election.cc:288] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:07:51.344365 14098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:51.344671 14098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [term 1 LEADER]: Becoming Leader. State: Replica: 3e45e00135db45e9a6ef9f91078269ca, State: Running, Role: LEADER
I20260620 09:07:51.345108 14098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } }
I20260620 09:07:51.345772 14093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:51.346630 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:51.331023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.0.98.62:33205)
0620 09:07:51.331537 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:51.331554 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:51.331593 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:07:51.332318 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:51.332338 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:51.332742 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:51.333157 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:51.334043 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:51.335476 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:51.336475 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:51.337234 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:51.337424 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:51.340334 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:51.340340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:51.340363 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:51.340404 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:51.342615 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:51.343138 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:51.343145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:51.343152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:51.343320 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:51.343649 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:51.343652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:51.343654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:51.344302 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:51.344532 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:51.345248 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 09:07:51.345571 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":168,"threads_started":1}
I20260620 09:07:51.347131 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e45e00135db45e9a6ef9f91078269ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } }
I20260620 09:07:51.347139 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e45e00135db45e9a6ef9f91078269ca. Latest consensus state: current_term: 1 leader_uuid: "3e45e00135db45e9a6ef9f91078269ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e45e00135db45e9a6ef9f91078269ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } }
I20260620 09:07:51.347289 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:51.347289 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:51.347865 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:51.350416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:07:51.351635 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:51.357769 14107 catalog_manager.cc:1347] Generated new cluster ID: 486c25f34ab448c0aaddc8f658c05f6b
I20260620 09:07:51.357873 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:51.370069 14107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:07:51.371105 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:51.377524 14107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e45e00135db45e9a6ef9f91078269ca: Generated new TSK 0
I20260620 09:07:51.378268 14107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:07:51.406621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14025
2026-06-20T09:07:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1525 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14120](info): set up 2 sockets
Jun 20 09:07:51 dist-test-slave-92t4 krb5kdc[14120](info): commencing operation
krb5kdc: starting...
W20260620 09:07:52.449332 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:07:52 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:07:52.575798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:52.692528 14133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:52.692809 14133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:52.692888 14133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:52.696975 14133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:52.697070 14133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:52.697129 14133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:52.697170 14133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:52.697206 14133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:52.702860 14133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:52.704090 14133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:52.705222 14133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:52.713815 14138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:52.713896 14133 server_base.cc:1034] running on GCE node
W20260620 09:07:52.713819 14139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:52.713816 14141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:52.714704 14133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:52.715852 14133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:52.717087 14133 hybrid_clock.cc:648] HybridClock initialized: now 1781946472717075 us; error 37 us; skew 500 ppm
Jun 20 09:07:52 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:52.723699 14133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:52.724725 14133 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260620 09:07:52.725414 14133 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:52.725528 14133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:52.725806 14133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:52.728063 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "90915675417c4424b04b16bc5d70d638"
format_stamp: "Formatted at 2026-06-20 09:07:52 on dist-test-slave-92t4"
server_key: "4aa8effe06fe17fa009ae915b82e8289"
server_key_iv: "0d887f4440415a2e1e2aa76e1918d114"
server_key_version: "encryptionkey@0"
I20260620 09:07:52.728658 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "90915675417c4424b04b16bc5d70d638"
format_stamp: "Formatted at 2026-06-20 09:07:52 on dist-test-slave-92t4"
server_key: "4aa8effe06fe17fa009ae915b82e8289"
server_key_iv: "0d887f4440415a2e1e2aa76e1918d114"
server_key_version: "encryptionkey@0"
I20260620 09:07:52.732610 14133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:07:52.735257 14148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:52.736579 14133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 09:07:52.736727 14133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "90915675417c4424b04b16bc5d70d638"
format_stamp: "Formatted at 2026-06-20 09:07:52 on dist-test-slave-92t4"
server_key: "4aa8effe06fe17fa009ae915b82e8289"
server_key_iv: "0d887f4440415a2e1e2aa76e1918d114"
server_key_version: "encryptionkey@0"
I20260620 09:07:52.736845 14133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:52.758363 14133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:52.761766 14133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:52.762015 14133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:52.769290 14133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33539
I20260620 09:07:52.769608 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33539 every 8 connection(s)
I20260620 09:07:52.770328 14133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:52.773183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14133
I20260620 09:07:52.773298 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:52.773321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:07:52.773658 392 external_mini_cluster.cc:1428] Setting key 6082c5d42cd43dd02ab0c33f9204a8a3
I20260620 09:07:52.780654 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap starting.
Jun 20 09:07:52 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:52.783494 14201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:52.784438 14201 log.cc:824] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:52.786491 14201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: No bootstrap required, opened a new log
I20260620 09:07:52.788303 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:52.775035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.0.98.62:33539)
0620 09:07:52.775449 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:52.775468 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:52.775505 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:07:52.776210 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:52.776229 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:52.776569 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:52.776852 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:52.777790 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:52.778773 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:52.779959 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:52.780437 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:52.780554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:52.783391 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:52.783396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:52.783409 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:52.783439 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:52.785439 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:52.785906 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:52.785917 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:52.785922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:52.786040 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:52.786339 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:52.786343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:52.786345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:52.786785 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:52.786991 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:52.787295 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 09:07:52.787586 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":135,"threads_started":1}
I20260620 09:07:52.790628 14201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:52.790916 14201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:52.790978 14201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Initialized, Role: FOLLOWER
I20260620 09:07:52.791426 14201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:52.791579 14201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:52.791646 14201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:52.791740 14201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:52.792690 14201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:52.793092 14201 leader_election.cc:302] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90915675417c4424b04b16bc5d70d638; no voters:
I20260620 09:07:52.793473 14201 leader_election.cc:288] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:07:52.793582 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:52.793824 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 LEADER]: Becoming Leader. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Running, Role: LEADER
I20260620 09:07:52.794131 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:52.794751 14201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:52.795806 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:52.795810 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90915675417c4424b04b16bc5d70d638. Latest consensus state: current_term: 1 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:52.795966 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:52.795966 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:52.796330 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:52.799604 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:52.806095 14215 catalog_manager.cc:1347] Generated new cluster ID: 59900dc06f314e09aef34041c81af6d3
I20260620 09:07:52.806178 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:52.817178 14215 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:07:52.818502 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:52.831336 14215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Generated new TSK 0
I20260620 09:07:52.832247 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:07:52.867802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:52.982084 14227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:52.982353 14227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:52.982435 14227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:52.986582 14227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:52.986649 14227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:52.986768 14227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:07:52.992266 14227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:52.993486 14227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:52.994629 14227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:53.004745 14233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.004747 14232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.004747 14235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:53.005223 14227 server_base.cc:1034] running on GCE node
I20260620 09:07:53.005621 14227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:53.006258 14227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:53.007426 14227 hybrid_clock.cc:648] HybridClock initialized: now 1781946473007410 us; error 29 us; skew 500 ppm
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:53.010334 14227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:07:53.011309 14227 webserver.cc:460] Webserver started at http://127.0.98.1:46865/ using document root <none> and password file <none>
I20260620 09:07:53.011894 14227 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:53.011957 14227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:53.012197 14227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:53.014415 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "bbb2001f29684f74b61565f85c61f1ec"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "6b54b67a334b1cdca8d0d5afd3c612c9"
server_key_iv: "f89b9ff338aa55823a3d435ea8555270"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.015008 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb2001f29684f74b61565f85c61f1ec"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "6b54b67a334b1cdca8d0d5afd3c612c9"
server_key_iv: "f89b9ff338aa55823a3d435ea8555270"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.020081 14227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 09:07:53.022698 14242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.024034 14227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:07:53.024161 14227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "bbb2001f29684f74b61565f85c61f1ec"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "6b54b67a334b1cdca8d0d5afd3c612c9"
server_key_iv: "f89b9ff338aa55823a3d435ea8555270"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.024292 14227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:53.039157 14227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:53.042549 14227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:53.042771 14227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:53.043417 14227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:53.044374 14227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:53.044445 14227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.044523 14227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:53.044566 14227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.054006 14227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38199
I20260620 09:07:53.054030 14355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38199 every 8 connection(s)
I20260620 09:07:53.055116 14227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:07:53.064751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14227
I20260620 09:07:53.064870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:53.065243 392 external_mini_cluster.cc:1428] Setting key 417e9c50196136f682faff85f9ec38e3
I20260620 09:07:53.069747 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.057036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53737 (local address 127.0.98.62:33539)
0620 09:07:53.057216 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.057220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.058086 (+ 866us) server_negotiation.cc:407] Connection header received
0620 09:07:53.059427 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.059430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.059489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.059573 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:53.060810 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.061457 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.062646 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.062901 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.063025 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.065707 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:53.065710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:53.065712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:53.065740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:53.067483 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.068054 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.068058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.068059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.068150 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.068516 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.068520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.068522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.068722 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:53.068838 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.069435 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.069571 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:07:53.070674 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.057472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:53737)
0620 09:07:53.057933 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.057977 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.059197 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.059750 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.059756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.060137 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:53.060652 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.060666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.061602 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.061611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.062512 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.062521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.063078 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.063082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.063255 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.063841 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:53.063858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:53.065503 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:53.067620 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.067627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.067645 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.067936 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.068246 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.068250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.068252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.068414 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.068834 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:53.068841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:53.069190 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.069395 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.069778 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260620 09:07:53.072031 14356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:53.072320 14356 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:53.072860 14356 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:07:53.074782 14165 ts_manager.cc:194] Registered new tserver with Master: bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199)
I20260620 09:07:53.076076 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:07:53.092702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:53.210759 14361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:53.211050 14361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:53.211095 14361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:53.215317 14361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:53.215399 14361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:53.215461 14361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:07:53.221015 14361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:53.222235 14361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:53.223344 14361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:53.233377 14367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.233376 14366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.233377 14369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:53.233671 14361 server_base.cc:1034] running on GCE node
I20260620 09:07:53.234187 14361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:53.234848 14361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:53.236066 14361 hybrid_clock.cc:648] HybridClock initialized: now 1781946473236022 us; error 54 us; skew 500 ppm
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:53.239089 14361 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:07:53.240166 14361 webserver.cc:460] Webserver started at http://127.0.98.2:34019/ using document root <none> and password file <none>
I20260620 09:07:53.240808 14361 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:53.240916 14361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:53.241196 14361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:53.243397 14361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "78de4fb39885efdecc72300fb4df79e9"
server_key_iv: "b4ee2d7d9d433c1764a17bb91ded3f1c"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.244001 14361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "78de4fb39885efdecc72300fb4df79e9"
server_key_iv: "b4ee2d7d9d433c1764a17bb91ded3f1c"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.248253 14361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:07:53.251011 14376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.252300 14361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:07:53.252454 14361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "78de4fb39885efdecc72300fb4df79e9"
server_key_iv: "b4ee2d7d9d433c1764a17bb91ded3f1c"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.252576 14361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:53.267769 14361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:53.271029 14361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:53.271248 14361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:53.271858 14361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:53.272713 14361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:53.272787 14361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.272858 14361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:53.272920 14361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.282028 14361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39333
I20260620 09:07:53.282073 14489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39333 every 8 connection(s)
I20260620 09:07:53.282997 14361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:07:53.290076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14361
I20260620 09:07:53.290223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:07:53.290501 392 external_mini_cluster.cc:1428] Setting key 52f46599b2afc5f4e6581a259ef553c3
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:53.298707 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.284951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35595 (local address 127.0.98.62:33539)
0620 09:07:53.285127 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.285130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.286214 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 09:07:53.287536 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.287539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.287601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.287688 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:53.288916 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.289533 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.290835 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.291077 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.291190 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.294048 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:53.294054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:53.294059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:53.294091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:53.296267 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.296931 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.296936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.296939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.297061 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.297403 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.297410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.297414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.297715 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:53.297834 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.298337 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.298491 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:07:53.299468 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.285478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:35595)
0620 09:07:53.286022 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.286078 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.287302 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.287883 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.287891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.288256 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:53.288759 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.288772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.289664 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.289668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.290704 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.290714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.291230 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.291234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.291418 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.292060 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:53.292077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:53.293852 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:53.296397 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.296418 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.296431 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.296820 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.297162 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.297165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.297166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.297301 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.297826 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:53.297831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:53.298085 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.298247 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.298517 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:07:53.300771 14490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:53.301080 14490 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:53.301612 14490 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
I20260620 09:07:53.302745 14165 ts_manager.cc:194] Registered new tserver with Master: 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
I20260620 09:07:53.303382 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:07:53.320494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:53.434260 14495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:53.434592 14495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:53.434670 14495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:53.438797 14495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:53.438912 14495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:53.439007 14495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:07:53.444805 14495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:53.446107 14495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:53.447212 14495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:53.457183 14500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.457183 14501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:53.457233 14495 server_base.cc:1034] running on GCE node
W20260620 09:07:53.457187 14503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:53.457773 14495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:53.458390 14495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:53.459563 14495 hybrid_clock.cc:648] HybridClock initialized: now 1781946473459545 us; error 28 us; skew 500 ppm
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:53.462436 14495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:07:53.463379 14495 webserver.cc:460] Webserver started at http://127.0.98.3:38919/ using document root <none> and password file <none>
I20260620 09:07:53.463966 14495 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:53.464071 14495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:53.464309 14495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:07:53.466562 14495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "814c15fa513c4b7ba3a33922726009a0"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "d8805effdaf1922914662a0b11d4c886"
server_key_iv: "553457977d341419e4ed667f2c07e690"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.467171 14495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "814c15fa513c4b7ba3a33922726009a0"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "d8805effdaf1922914662a0b11d4c886"
server_key_iv: "553457977d341419e4ed667f2c07e690"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.471189 14495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:07:53.473732 14510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.475028 14495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:07:53.475178 14495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "814c15fa513c4b7ba3a33922726009a0"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "d8805effdaf1922914662a0b11d4c886"
server_key_iv: "553457977d341419e4ed667f2c07e690"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.475296 14495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:53.496095 14495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:53.499415 14495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:53.499642 14495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:53.500277 14495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:53.501230 14495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:53.501299 14495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.501371 14495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:53.501422 14495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.510854 14495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33355
I20260620 09:07:53.510880 14623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33355 every 8 connection(s)
I20260620 09:07:53.511898 14495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:07:53.517614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14495
I20260620 09:07:53.517724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:07:53.517990 392 external_mini_cluster.cc:1428] Setting key f2aa74d5f0dbb8033e4c00213bfee2ac
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:53.526378 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.513980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48907 (local address 127.0.98.62:33539)
0620 09:07:53.514127 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.514132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.515096 (+ 964us) server_negotiation.cc:407] Connection header received
0620 09:07:53.516447 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.516450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.516509 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.516591 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:53.517952 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.518711 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.520089 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.520341 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.520456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.523130 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:53.523133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:53.523135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:53.523162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:53.524395 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.524863 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.524865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.524866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.524939 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.525297 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.525299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.525300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.525484 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:53.525569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.526099 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.526207 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:07:53.527220 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.514415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:48907)
0620 09:07:53.514932 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.514979 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.516218 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.516772 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.516780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.517235 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:53.517807 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.517823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.518848 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.518852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.519963 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.519974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.520506 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.520510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.520695 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.521395 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:53.521412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:53.522934 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:53.524503 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.524508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.524522 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.524777 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.525016 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:53.525018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:53.525019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:53.525217 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:53.525574 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:53.525579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:53.525858 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.526037 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.526310 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260620 09:07:53.528481 14624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:53.528743 14624 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:53.529307 14624 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
I20260620 09:07:53.530423 14165 ts_manager.cc:194] Registered new tserver with Master: 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355)
I20260620 09:07:53.531177 14165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48907
I20260620 09:07:53.532230 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:07:53.541239 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.534222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.0.98.62:33539)
0620 09:07:53.534337 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.534342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.534509 (+ 167us) server_negotiation.cc:407] Connection header received
0620 09:07:53.534606 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.534608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.534644 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.534709 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:53.535372 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.535866 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.536637 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.536837 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.536910 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.538308 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:53.538312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:53.538314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:53.538333 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:53.539577 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.539948 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.539950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.539952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.540022 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:53.540259 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:53.540261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:53.540263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:53.540529 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:53.540642 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.540927 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.541022 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:07:53.543977 14165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:07:53.547093 14165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:07:53.561995 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.555870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39333 (local address 127.0.0.1:38414)
0620 09:07:53.556553 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.556607 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.556804 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.557625 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.557630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.557654 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:53.557785 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.557792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.559001 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.559003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.560156 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.560165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.561423 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.561426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.561534 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.561576 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.561694 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.561814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":98,"threads_started":1}
I20260620 09:07:53.562023 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.556018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.98.2:39333)
0620 09:07:53.556483 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.556496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.556701 (+ 205us) server_negotiation.cc:407] Connection header received
0620 09:07:53.557007 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.557014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.557269 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.557398 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:53.557929 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.558862 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.560284 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.561316 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.561427 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.561586 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.561751 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.561858 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":127,"threads_started":1}
I20260620 09:07:53.562206 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.556014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33355 (local address 127.0.0.1:40646)
0620 09:07:53.556661 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.556677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.556804 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.558157 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.558160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.558177 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:53.558278 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.558284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.559598 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.559601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.560711 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.560720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.561813 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.561817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.561917 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.561941 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.561989 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.562068 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":117,"threads_started":1}
I20260620 09:07:53.562716 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.556692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38199 (local address 127.0.0.1:55446)
0620 09:07:53.557072 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.557084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.557160 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.557946 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.557949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.557962 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:53.558078 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.558088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.560303 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.560306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.561259 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.561267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.562340 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.562343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.562436 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.562456 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.562533 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.562598 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":109,"threads_started":1}
I20260620 09:07:53.562754 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.557241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.0.98.3:33355)
0620 09:07:53.557698 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.557712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.557735 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:07:53.557817 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.557823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.558008 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.558188 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:53.558381 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.559448 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.560836 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.561654 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.562313 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.562458 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.562510 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.562614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":161,"threads_started":1}
I20260620 09:07:53.563005 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.556861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.98.1:38199)
0620 09:07:53.557491 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.557505 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.557532 (+ 27us) server_negotiation.cc:407] Connection header received
0620 09:07:53.557611 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.557615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.557790 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.557976 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:53.558216 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.559436 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.561386 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.562212 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.562317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.562464 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.562765 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.562870 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":101,"threads_started":1}
I20260620 09:07:53.565608 14558 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c (DEFAULT_TABLE table=test-table [id=da6c3f6dd56f4cd49cab374521e0adef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:53.565665 14290 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c (DEFAULT_TABLE table=test-table [id=da6c3f6dd56f4cd49cab374521e0adef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:53.565665 14424 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c (DEFAULT_TABLE table=test-table [id=da6c3f6dd56f4cd49cab374521e0adef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:53.566773 14424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8401db872c4df0afaada31ae6e4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:53.566773 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8401db872c4df0afaada31ae6e4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:53.566794 14558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8401db872c4df0afaada31ae6e4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:53.573331 14641 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Bootstrap starting.
I20260620 09:07:53.574712 14642 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Bootstrap starting.
I20260620 09:07:53.575630 14641 tablet_bootstrap.cc:654] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:53.576122 14643 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Bootstrap starting.
I20260620 09:07:53.576613 14642 tablet_bootstrap.cc:654] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:53.576983 14641 log.cc:824] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:53.577461 14642 log.cc:824] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:53.578924 14643 tablet_bootstrap.cc:654] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:53.579319 14641 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: No bootstrap required, opened a new log
I20260620 09:07:53.579370 14642 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: No bootstrap required, opened a new log
I20260620 09:07:53.579530 14642 ts_tablet_manager.cc:1397] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:07:53.579540 14641 ts_tablet_manager.cc:1397] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:07:53.579970 14643 log.cc:824] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:53.581836 14643 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: No bootstrap required, opened a new log
I20260620 09:07:53.582057 14643 ts_tablet_manager.cc:1397] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:07:53.583778 14641 raft_consensus.cc:348] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.584120 14641 raft_consensus.cc:374] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:53.583959 14642 raft_consensus.cc:348] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.584218 14642 raft_consensus.cc:374] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:53.584245 14641 raft_consensus.cc:729] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c15fa513c4b7ba3a33922726009a0, State: Initialized, Role: FOLLOWER
I20260620 09:07:53.584321 14642 raft_consensus.cc:729] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb2001f29684f74b61565f85c61f1ec, State: Initialized, Role: FOLLOWER
I20260620 09:07:53.584817 14641 consensus_queue.cc:260] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.584817 14642 consensus_queue.cc:260] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.585212 14643 raft_consensus.cc:348] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.585547 14643 raft_consensus.cc:374] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:53.585642 14643 raft_consensus.cc:729] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa17f1d76f49a9a83d18dcce508c9d, State: Initialized, Role: FOLLOWER
I20260620 09:07:53.585963 14642 ts_tablet_manager.cc:1428] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 09:07:53.585963 14641 ts_tablet_manager.cc:1428] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:07:53.586213 14643 consensus_queue.cc:260] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.586345 14624 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
I20260620 09:07:53.586367 14356 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
I20260620 09:07:53.586884 14490 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
I20260620 09:07:53.587221 14643 ts_tablet_manager.cc:1428] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:07:53.691445 14647 raft_consensus.cc:482] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:53.691672 14647 raft_consensus.cc:504] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.692842 14647 leader_election.cc:288] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333), 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355)
I20260620 09:07:53.697363 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.693251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59227 (local address 127.0.98.3:33355)
0620 09:07:53.693379 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.693384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.693675 (+ 291us) server_negotiation.cc:407] Connection header received
0620 09:07:53.693783 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.693786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.693848 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.693929 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:53.694273 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.694795 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696176 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.696624 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696773 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.696843 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.697133 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.697222 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:07:53.697381 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.693245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42131 (local address 127.0.98.2:39333)
0620 09:07:53.693379 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.693384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.693811 (+ 427us) server_negotiation.cc:407] Connection header received
0620 09:07:53.693880 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.693882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.693967 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.694052 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:53.694327 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.694801 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696251 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.696662 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696790 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.696856 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.697186 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.697272 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:07:53.697397 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.693176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39333 (local address 127.0.98.1:42131)
0620 09:07:53.693673 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.693691 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.693801 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.694080 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.694082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.694095 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:53.694187 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.694192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.694961 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.694964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.696065 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.696075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.696795 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.696938 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.696985 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.697147 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.697223 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":141,"thread_start_us":95,"threads_started":1}
I20260620 09:07:53.697407 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.693176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33355 (local address 127.0.98.1:59227)
0620 09:07:53.693563 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.693576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:53.693689 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:53.693956 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:53.693959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:53.693975 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:53.694147 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.694159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.694915 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.694918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.696037 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:53.696047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.696783 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.696786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:53.696935 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.696968 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 09:07:53.697098 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 09:07:53.697211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":111,"thread_start_us":143,"threads_started":1}
I20260620 09:07:53.698170 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8401db872c4df0afaada31ae6e4c7c" candidate_uuid: "bbb2001f29684f74b61565f85c61f1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" is_pre_election: true
I20260620 09:07:53.698239 14578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8401db872c4df0afaada31ae6e4c7c" candidate_uuid: "bbb2001f29684f74b61565f85c61f1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c15fa513c4b7ba3a33922726009a0" is_pre_election: true
I20260620 09:07:53.698539 14444 raft_consensus.cc:2454] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb2001f29684f74b61565f85c61f1ec in term 0.
I20260620 09:07:53.698549 14578 raft_consensus.cc:2454] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb2001f29684f74b61565f85c61f1ec in term 0.
I20260620 09:07:53.699074 14243 leader_election.cc:302] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0, bbb2001f29684f74b61565f85c61f1ec; no voters:
I20260620 09:07:53.699374 14647 raft_consensus.cc:2781] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:07:53.699476 14647 raft_consensus.cc:482] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:07:53.699542 14647 raft_consensus.cc:3037] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:53.700641 14647 raft_consensus.cc:504] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.701081 14647 leader_election.cc:288] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 1 election: Requested vote from peers 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333), 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355)
I20260620 09:07:53.701501 14444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8401db872c4df0afaada31ae6e4c7c" candidate_uuid: "bbb2001f29684f74b61565f85c61f1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
I20260620 09:07:53.701501 14578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8401db872c4df0afaada31ae6e4c7c" candidate_uuid: "bbb2001f29684f74b61565f85c61f1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814c15fa513c4b7ba3a33922726009a0"
I20260620 09:07:53.701642 14444 raft_consensus.cc:3037] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:53.701642 14578 raft_consensus.cc:3037] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:53.702999 14578 raft_consensus.cc:2454] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb2001f29684f74b61565f85c61f1ec in term 1.
I20260620 09:07:53.703004 14444 raft_consensus.cc:2454] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb2001f29684f74b61565f85c61f1ec in term 1.
I20260620 09:07:53.703290 14243 leader_election.cc:302] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814c15fa513c4b7ba3a33922726009a0, bbb2001f29684f74b61565f85c61f1ec; no voters:
I20260620 09:07:53.703464 14647 raft_consensus.cc:2781] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:53.703701 14647 raft_consensus.cc:686] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 LEADER]: Becoming Leader. State: Replica: bbb2001f29684f74b61565f85c61f1ec, State: Running, Role: LEADER
I20260620 09:07:53.704099 14647 consensus_queue.cc:237] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:53.707701 14165 catalog_manager.cc:5495] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec reported cstate change: term changed from 0 to 1, leader changed from <none> to bbb2001f29684f74b61565f85c61f1ec (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bbb2001f29684f74b61565f85c61f1ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } health_report { overall_health: HEALTHY } } }
I20260620 09:07:53.753949 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.750229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.0.98.1:38199)
0620 09:07:53.750385 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:53.750392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:53.750411 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:07:53.750506 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:53.750508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:53.750566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:53.750685 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:07:53.750952 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.751464 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.752343 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:53.752583 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:53.752698 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:53.753084 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:07:53.753536 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:07:53.753683 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:53.753720 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 09:07:53.753786 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:07:53.760450 14578 raft_consensus.cc:1261] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Refusing update from remote peer bbb2001f29684f74b61565f85c61f1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:53.760450 14444 raft_consensus.cc:1261] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Refusing update from remote peer bbb2001f29684f74b61565f85c61f1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:53.761430 14647 consensus_queue.cc:1035] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:53.761722 14652 consensus_queue.cc:1035] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:07:53.762879 14625 tablet.cc:2369] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:07:53.781097 14163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:53.781291 14163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:53.783722 14163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Sending DeleteTablet for 3 replicas of tablet 5e8401db872c4df0afaada31ae6e4c7c
I20260620 09:07:53.785085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14227
I20260620 09:07:53.784998 14424 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260620 09:07:53.786053 14665 tablet_replica.cc:330] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: stopping tablet replica
I20260620 09:07:53.786417 14665 raft_consensus.cc:2229] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:07:53.786773 14665 raft_consensus.cc:2258] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:53.787417 14558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40646
I20260620 09:07:53.787956 14666 tablet_replica.cc:330] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: stopping tablet replica
I20260620 09:07:53.788262 14665 ts_tablet_manager.cc:1905] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:53.788297 14666 raft_consensus.cc:2229] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:07:53.788610 14666 raft_consensus.cc:2258] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:53.790665 14665 ts_tablet_manager.cc:1918] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:53.790735 14666 ts_tablet_manager.cc:1905] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:53.790788 14665 log.cc:1196] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/5e8401db872c4df0afaada31ae6e4c7c
I20260620 09:07:53.791079 14665 ts_tablet_manager.cc:1939] T 5e8401db872c4df0afaada31ae6e4c7c P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting consensus metadata
I20260620 09:07:53.792233 14151 catalog_manager.cc:4841] TS 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): tablet 5e8401db872c4df0afaada31ae6e4c7c (table test-table [id=da6c3f6dd56f4cd49cab374521e0adef]) successfully deleted
W20260620 09:07:53.793673 14149 connection.cc:535] client connection to 127.0.98.1:38199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 09:07:53.793829 14149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:07:53.793999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14361
I20260620 09:07:53.794055 14666 ts_tablet_manager.cc:1918] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:53.794178 14666 log.cc:1196] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/5e8401db872c4df0afaada31ae6e4c7c
I20260620 09:07:53.794466 14666 ts_tablet_manager.cc:1939] T 5e8401db872c4df0afaada31ae6e4c7c P 814c15fa513c4b7ba3a33922726009a0: Deleting consensus metadata
I20260620 09:07:53.795687 14149 catalog_manager.cc:4841] TS 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): tablet 5e8401db872c4df0afaada31ae6e4c7c (table test-table [id=da6c3f6dd56f4cd49cab374521e0adef]) successfully deleted
I20260620 09:07:53.796416 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:53.796111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38199 (local address 127.0.0.1:55462)
0620 09:07:53.796242 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:53.796331 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 09:07:53.796640 14149 catalog_manager.cc:4584] TS bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e8401db872c4df0afaada31ae6e4c7c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38199: connect: Connection refused (error 111)
I20260620 09:07:53.801453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14495
I20260620 09:07:53.807224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14133
I20260620 09:07:53.814023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--webserver_interface=127.0.98.62
--webserver_port=43883
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:53.929647 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:53.929975 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:53.930037 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:53.934424 14668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:53.934492 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:53.934512 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:53.934537 14668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:53.934553 14668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:53.940212 14668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:53.941380 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:53.942400 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:53.955937 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.955937 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:53.955979 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:53.956317 14668 server_base.cc:1034] running on GCE node
I20260620 09:07:53.957079 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:53.958393 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:53.959623 14668 hybrid_clock.cc:648] HybridClock initialized: now 1781946473959582 us; error 57 us; skew 500 ppm
Jun 20 09:07:53 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:53.962654 14668 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:53.963658 14668 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260620 09:07:53.964280 14668 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:53.964397 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:53.968699 14668 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:53.971287 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:53.972775 14668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 09:07:53.972968 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "90915675417c4424b04b16bc5d70d638"
format_stamp: "Formatted at 2026-06-20 09:07:52 on dist-test-slave-92t4"
server_key: "4aa8effe06fe17fa009ae915b82e8289"
server_key_iv: "0d887f4440415a2e1e2aa76e1918d114"
server_key_version: "encryptionkey@0"
I20260620 09:07:53.973738 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:53.988080 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:53.991619 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:53.991874 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:53.999500 14668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33539
I20260620 09:07:53.999835 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33539 every 8 connection(s)
I20260620 09:07:54.000718 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:54.001129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14668
I20260620 09:07:54.001598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38199
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=46865
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:54.005726 14737 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:07:54.007616 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap starting.
I20260620 09:07:54.025975 14737 log.cc:824] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:54.035341 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:07:54.035815 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap complete.
I20260620 09:07:54.040069 14737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:54.040544 14737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Initialized, Role: FOLLOWER
I20260620 09:07:54.041191 14737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:54.041358 14737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:54.041458 14737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:54.041579 14737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:07:54.043728 14737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:54.044131 14737 leader_election.cc:302] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90915675417c4424b04b16bc5d70d638; no voters:
I20260620 09:07:54.044504 14737 leader_election.cc:288] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:07:54.044600 14741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:07:54.044826 14741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 LEADER]: Becoming Leader. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Running, Role: LEADER
I20260620 09:07:54.045172 14741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:54.045918 14737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:54.046216 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90915675417c4424b04b16bc5d70d638. Latest consensus state: current_term: 2 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:54.046347 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:54.046594 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:54.046720 14745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:54.046706 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:54.052066 14745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:54.052907 14745 catalog_manager.cc:1259] Loaded cluster ID: 59900dc06f314e09aef34041c81af6d3
I20260620 09:07:54.053004 14745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:07:54.054160 14745 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 09:07:54.056962 14745 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 09:07:54.062213 14745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:07:54.062958 14745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Loaded TSK: 0
I20260620 09:07:54.064024 14745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:07:54.133924 14739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:54.134378 14739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:54.134476 14739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:54.140658 14739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:54.140767 14739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:54.140877 14739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:07:54.146943 14739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46865
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:54.148336 14739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:54.149529 14739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:54.160866 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.161222 14739 server_base.cc:1034] running on GCE node
W20260620 09:07:54.160898 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:54.160892 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.161963 14739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:54.162669 14739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:54.163940 14739 hybrid_clock.cc:648] HybridClock initialized: now 1781946474163904 us; error 52 us; skew 500 ppm
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:54.166896 14739 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:07:54.167878 14739 webserver.cc:460] Webserver started at http://127.0.98.1:46865/ using document root <none> and password file <none>
I20260620 09:07:54.168499 14739 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:54.168578 14739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:54.173107 14739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:54.175623 14774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.177182 14739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:54.177335 14739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "bbb2001f29684f74b61565f85c61f1ec"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "6b54b67a334b1cdca8d0d5afd3c612c9"
server_key_iv: "f89b9ff338aa55823a3d435ea8555270"
server_key_version: "encryptionkey@0"
I20260620 09:07:54.177997 14739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:54.199209 14739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:54.202665 14739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:54.202891 14739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:54.203588 14739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:54.204825 14781 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:07:54.209249 14739 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:07:54.209321 14739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 09:07:54.209373 14739 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:07:54.211092 14739 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:07:54.211155 14739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 09:07:54.211422 14781 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Bootstrap starting.
I20260620 09:07:54.222389 14739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38199
I20260620 09:07:54.222445 14888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38199 every 8 connection(s)
I20260620 09:07:54.223812 14739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:07:54.229212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14739
I20260620 09:07:54.229827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39333
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=34019
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:54.233464 14781 log.cc:824] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Log is configured to *not* fsync() on all Append() calls
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:54.242671 14781 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:07:54.243137 14781 tablet_bootstrap.cc:492] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Bootstrap complete.
I20260620 09:07:54.243649 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.226239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:40623)
0620 09:07:54.226775 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.226817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.227698 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.228863 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.228871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.229380 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:54.229884 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.229925 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.231338 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.231345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.232848 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.232865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.233949 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.233961 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.234192 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.234913 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:54.234935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:54.237027 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:54.239805 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.239815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.239834 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.240244 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.240704 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.240707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.240709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.240915 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.241679 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:54.241689 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:54.242052 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.242284 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.242644 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":167,"threads_started":1}
I20260620 09:07:54.244134 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.226185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40623 (local address 127.0.98.62:33539)
0620 09:07:54.226565 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.226587 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.226948 (+ 361us) server_negotiation.cc:407] Connection header received
0620 09:07:54.228132 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.228144 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.228441 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.228783 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:54.230132 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.231160 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.232991 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.233765 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.233947 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.237320 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:54.237329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:54.237350 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:54.237386 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:54.239652 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.240392 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.240399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.240404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.240585 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.241031 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.241034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.241061 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.241545 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:54.241680 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.242435 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.242871 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260620 09:07:54.244553 14781 ts_tablet_manager.cc:1397] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.010s
I20260620 09:07:54.246893 14889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:54.247141 14889 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:54.247714 14889 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
I20260620 09:07:54.247442 14781 raft_consensus.cc:348] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.247946 14781 raft_consensus.cc:729] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb2001f29684f74b61565f85c61f1ec, State: Initialized, Role: FOLLOWER
I20260620 09:07:54.248508 14781 consensus_queue.cc:260] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.249590 14781 ts_tablet_manager.cc:1428] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260620 09:07:54.251425 14701 ts_manager.cc:194] Registered new tserver with Master: bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199)
I20260620 09:07:54.255575 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40623
I20260620 09:07:54.258272 14889 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:07:54.265964 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.254929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.98.1:38199)
0620 09:07:54.255940 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.255951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.255976 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:07:54.256034 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.256038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.256194 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.256344 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:54.257097 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.258250 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.259347 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.259698 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.259820 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.262298 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:54.262316 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:54.262325 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:54.262348 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:54.263895 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.264413 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.264417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.264419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.264496 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.264773 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.264778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.264780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.265032 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:54.265237 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.265603 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.265767 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":914,"thread_start_us":108,"threads_started":1}
I20260620 09:07:54.265955 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.254874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38199 (local address 127.0.0.1:55476)
0620 09:07:54.255533 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.255586 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.255734 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.256347 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.256353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.256797 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:54.256931 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.256940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.258385 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.258389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.259225 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.259235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.259808 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.259826 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.259945 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.260555 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:54.260567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:54.262160 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:54.264038 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.264049 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.264054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.264286 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.264591 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.264595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.264597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.264684 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.265221 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:54.265228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:54.265374 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.265524 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.265743 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":157,"threads_started":1}
I20260620 09:07:54.267094 14823 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8401db872c4df0afaada31ae6e4c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55476
I20260620 09:07:54.268081 14898 tablet_replica.cc:330] stopping tablet replica
I20260620 09:07:54.268385 14898 raft_consensus.cc:2229] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:07:54.268662 14898 raft_consensus.cc:2258] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:54.269709 14898 ts_tablet_manager.cc:1905] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:54.273303 14898 ts_tablet_manager.cc:1918] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:54.273423 14898 log.cc:1196] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/5e8401db872c4df0afaada31ae6e4c7c
I20260620 09:07:54.273695 14898 ts_tablet_manager.cc:1939] T 5e8401db872c4df0afaada31ae6e4c7c P bbb2001f29684f74b61565f85c61f1ec: Deleting consensus metadata
I20260620 09:07:54.275418 14685 catalog_manager.cc:4841] TS bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199): tablet 5e8401db872c4df0afaada31ae6e4c7c (table test-table [id=da6c3f6dd56f4cd49cab374521e0adef]) successfully deleted
W20260620 09:07:54.356094 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:54.356387 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:54.356438 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:54.360818 14893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:54.360905 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:54.360977 14893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:07:54.366883 14893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34019
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:54.368302 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:54.369637 14893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:54.380172 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.380420 14893 server_base.cc:1034] running on GCE node
W20260620 09:07:54.380229 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:54.380168 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.380985 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:54.381798 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:54.383013 14893 hybrid_clock.cc:648] HybridClock initialized: now 1781946474382985 us; error 47 us; skew 500 ppm
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:54.385993 14893 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:07:54.387027 14893 webserver.cc:460] Webserver started at http://127.0.98.2:34019/ using document root <none> and password file <none>
I20260620 09:07:54.387611 14893 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:54.387686 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:54.391872 14893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:07:54.394333 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.395735 14893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:54.395846 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "78de4fb39885efdecc72300fb4df79e9"
server_key_iv: "b4ee2d7d9d433c1764a17bb91ded3f1c"
server_key_version: "encryptionkey@0"
I20260620 09:07:54.396436 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:54.421465 14893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:54.426147 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:54.426322 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:54.426973 14893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:54.428215 14893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:54.428290 14893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.428359 14893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:54.428396 14893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.441241 14893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39333
I20260620 09:07:54.441385 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39333 every 8 connection(s)
I20260620 09:07:54.442691 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:07:54.448822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 14893
I20260620 09:07:54.449409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33355
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=38919
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:54.461928 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.445222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41873 (local address 127.0.98.62:33539)
0620 09:07:54.445370 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.445375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.446548 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 09:07:54.448358 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.448362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.448453 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.448591 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:54.450611 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.451390 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.453110 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.453351 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.453476 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.456972 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:54.456977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:54.456984 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:54.457022 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:54.458790 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.459537 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.459541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.459543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.459649 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.460157 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.460162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.460166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.460476 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:54.460569 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.461484 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.461668 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:07:54.463680 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.445713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:41873)
0620 09:07:54.446366 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.446427 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.448058 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.448959 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.448969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.449595 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:54.450377 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.450424 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.451540 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.451543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.452926 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.452941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.453524 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.453528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.453780 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.454553 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:54.454583 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:54.456727 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:54.458929 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.458939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.458961 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.459378 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.459785 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.459791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.459795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.460024 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.460666 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:54.460675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:54.461125 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.462143 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.462558 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":167,"threads_started":1}
I20260620 09:07:54.465104 15027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:54.465380 15027 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:54.466066 15027 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
I20260620 09:07:54.467305 14701 ts_manager.cc:194] Registered new tserver with Master: 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
I20260620 09:07:54.467964 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41873
W20260620 09:07:54.572852 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:54.573235 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:54.573318 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:54.577620 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:54.577713 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:54.577847 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:07:54.583824 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38919
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:54.584975 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:54.586112 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:54.596637 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:54.596575 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.596715 15030 server_base.cc:1034] running on GCE node
W20260620 09:07:54.596604 15038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:54.597294 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:54.597972 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:54.599170 15030 hybrid_clock.cc:648] HybridClock initialized: now 1781946474599150 us; error 42 us; skew 500 ppm
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:54.602198 15030 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:07:54.603111 15030 webserver.cc:460] Webserver started at http://127.0.98.3:38919/ using document root <none> and password file <none>
I20260620 09:07:54.603660 15030 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:54.603734 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:54.607918 15030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:07:54.610126 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.611439 15030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:07:54.611608 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "814c15fa513c4b7ba3a33922726009a0"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "d8805effdaf1922914662a0b11d4c886"
server_key_iv: "553457977d341419e4ed667f2c07e690"
server_key_version: "encryptionkey@0"
I20260620 09:07:54.612165 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:54.635293 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:54.638681 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:54.638906 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:54.639564 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:54.640444 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:54.640491 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.640581 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:54.640620 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:54.650579 15030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33355
I20260620 09:07:54.650602 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33355 every 8 connection(s)
I20260620 09:07:54.651676 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:07:54.657406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15030
Jun 20 09:07:54 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:07:54.667713 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.658148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.98.62:33539)
0620 09:07:54.658392 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.658398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.658566 (+ 168us) server_negotiation.cc:407] Connection header received
0620 09:07:54.658734 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.658736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.658781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.658864 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:54.659764 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.660466 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.661614 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.661838 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.661917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.663568 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:54.663572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:54.663575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:54.663630 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:54.665885 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666370 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.666372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.666373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.666476 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666789 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.666791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.666792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.667014 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:54.667225 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.667422 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.667539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"spinlock_wait_cycles":32896,"thread_start_us":68,"threads_started":1}
I20260620 09:07:54.668154 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.653665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40265 (local address 127.0.98.62:33539)
0620 09:07:54.653825 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.653829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.654731 (+ 902us) server_negotiation.cc:407] Connection header received
0620 09:07:54.656081 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.656084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.656143 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.656260 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:54.657733 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.658389 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.660067 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.660395 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.660511 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.663569 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:54.663572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:54.663575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:54.663631 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:54.665843 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666480 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.666483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.666484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.666561 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666946 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:54.666949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:54.666951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:54.667153 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:54.667231 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.667892 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.668008 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":8576}
I20260620 09:07:54.669204 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.654105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:40265)
0620 09:07:54.654562 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.654607 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.655843 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.656455 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.656463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.656842 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:54.657567 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.657588 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.658536 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.658543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.659925 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.659935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.660578 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.660582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.660781 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.661497 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:54.661519 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:54.663410 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:54.665999 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.666024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.666375 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.666655 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:54.666658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:54.666659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:54.666848 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:54.667241 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:54.667247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:54.667605 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.667848 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.668203 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260620 09:07:54.670518 15159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33539
I20260620 09:07:54.670769 15159 heartbeater.cc:464] Registering TS with master...
I20260620 09:07:54.671274 15159 heartbeater.cc:511] Master 127.0.98.62:33539 requested a full tablet report, sending...
I20260620 09:07:54.672263 14701 ts_manager.cc:194] Registered new tserver with Master: 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355)
I20260620 09:07:54.672833 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40265
I20260620 09:07:54.682171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:07:54.687033 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.682907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.0.98.62:33539)
0620 09:07:54.683073 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.683077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.683256 (+ 179us) server_negotiation.cc:407] Connection header received
0620 09:07:54.683474 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.683477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.683590 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.683695 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:07:54.684033 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.684548 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.685553 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.685770 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.685873 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.686176 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:07:54.686616 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:07:54.686713 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.686854 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.686906 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 09:07:54.687772 14701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:07:54.691138 14701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:07:54.698912 14823 tablet_service.cc:1467] Processing CreateTablet for tablet ac79d8383e49484c8cc4111841da1040 (DEFAULT_TABLE table=test-table [id=f63283c398c6454388d317aeea56f3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:54.699623 14823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac79d8383e49484c8cc4111841da1040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:54.702571 14781 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Bootstrap starting.
I20260620 09:07:54.703507 14781 tablet_bootstrap.cc:654] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:54.703845 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.698044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39333 (local address 127.0.0.1:38424)
0620 09:07:54.698414 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.698431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.698530 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.699314 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.699317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.699337 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:54.699479 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.699485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.700924 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.700927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.702342 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.702356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.703327 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.703330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.703424 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.703471 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.703569 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.703656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":31,"spinlock_wait_cycles":2816,"thread_start_us":123,"threads_started":1}
I20260620 09:07:54.704020 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.697896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33355 (local address 127.0.0.1:40652)
0620 09:07:54.698048 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.698064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.698141 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.699346 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.699348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.699367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:54.699577 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.699585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.701179 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.701182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.702430 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.702442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.703537 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.703542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.703662 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.703696 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.703754 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.703856 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":157}
I20260620 09:07:54.704778 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.698312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.0.98.2:39333)
0620 09:07:54.698770 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.698786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.698810 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:07:54.698897 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.698903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.699148 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.699329 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:54.699639 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.700770 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.702495 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.703196 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.703327 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.703494 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.704510 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.704626 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":138,"threads_started":1}
I20260620 09:07:54.704895 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.698458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.0.98.3:33355)
0620 09:07:54.698956 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.698967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.698987 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:07:54.699054 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.699058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.699196 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.699478 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:54.699723 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.701001 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.702595 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.703374 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.704383 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.704570 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.704628 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.704738 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":144,"threads_started":1}
I20260620 09:07:54.705142 14781 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: No bootstrap required, opened a new log
I20260620 09:07:54.705236 14781 ts_tablet_manager.cc:1397] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:07:54.705689 14781 raft_consensus.cc:348] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.705842 14781 raft_consensus.cc:374] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:54.705897 14781 raft_consensus.cc:729] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb2001f29684f74b61565f85c61f1ec, State: Initialized, Role: FOLLOWER
I20260620 09:07:54.706034 14781 consensus_queue.cc:260] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.706288 14781 ts_tablet_manager.cc:1428] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:07:54.708087 15093 tablet_service.cc:1467] Processing CreateTablet for tablet ac79d8383e49484c8cc4111841da1040 (DEFAULT_TABLE table=test-table [id=f63283c398c6454388d317aeea56f3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:54.708083 14951 tablet_service.cc:1467] Processing CreateTablet for tablet ac79d8383e49484c8cc4111841da1040 (DEFAULT_TABLE table=test-table [id=f63283c398c6454388d317aeea56f3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:07:54.709280 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac79d8383e49484c8cc4111841da1040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:54.709308 15093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac79d8383e49484c8cc4111841da1040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:07:54.716367 15172 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Bootstrap starting.
I20260620 09:07:54.716473 15173 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Bootstrap starting.
I20260620 09:07:54.718868 15172 tablet_bootstrap.cc:654] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:54.719255 15173 tablet_bootstrap.cc:654] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Neither blocks nor log segments found. Creating new log.
I20260620 09:07:54.719985 15173 log.cc:824] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:54.719983 15172 log.cc:824] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:54.722067 15173 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: No bootstrap required, opened a new log
I20260620 09:07:54.722082 15172 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: No bootstrap required, opened a new log
I20260620 09:07:54.722285 15172 ts_tablet_manager.cc:1397] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 09:07:54.722295 15173 ts_tablet_manager.cc:1397] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260620 09:07:54.725215 14890 tablet.cc:2369] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:07:54.725769 15173 raft_consensus.cc:348] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.725828 15172 raft_consensus.cc:348] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.726083 15173 raft_consensus.cc:374] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:54.726078 15172 raft_consensus.cc:374] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:07:54.726156 15172 raft_consensus.cc:729] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c15fa513c4b7ba3a33922726009a0, State: Initialized, Role: FOLLOWER
I20260620 09:07:54.726148 15173 raft_consensus.cc:729] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa17f1d76f49a9a83d18dcce508c9d, State: Initialized, Role: FOLLOWER
I20260620 09:07:54.726626 15173 consensus_queue.cc:260] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.726626 15172 consensus_queue.cc:260] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.727380 15159 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
I20260620 09:07:54.727592 15027 heartbeater.cc:503] Master 127.0.98.62:33539 was elected leader, sending a full tablet report...
I20260620 09:07:54.727717 15172 ts_tablet_manager.cc:1428] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:07:54.727716 15173 ts_tablet_manager.cc:1428] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:07:54.766987 15176 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:54.767263 15176 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.768524 15176 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333), bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199)
I20260620 09:07:54.772794 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.768833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39333 (local address 127.0.98.3:49049)
0620 09:07:54.768989 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.769007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.769152 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.769559 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.769564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.769589 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:54.769728 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.769736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.770676 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.770680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.771548 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.771557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.772318 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.772322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.772450 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.772476 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.772562 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.772652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 09:07:54.772799 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.769088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49049 (local address 127.0.98.2:39333)
0620 09:07:54.769244 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.769248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.769265 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:07:54.769364 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.769367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.769423 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.769518 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:54.769950 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.770565 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.771678 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.772202 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.772283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.772338 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.772582 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.772668 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:07:54.773173 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.768938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56429 (local address 127.0.98.1:38199)
0620 09:07:54.769257 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.769264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.769570 (+ 306us) server_negotiation.cc:407] Connection header received
0620 09:07:54.769675 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.769680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.769750 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.769869 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:07:54.770205 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.770772 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.771993 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.772584 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.772680 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.772859 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.772929 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.773013 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
I20260620 09:07:54.773190 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.768833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38199 (local address 127.0.98.3:56429)
0620 09:07:54.769473 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.769487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:54.769579 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:54.769945 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:54.769949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:54.769968 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:07:54.770083 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.770089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.770928 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.770930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.771860 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:54.771870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.772699 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.772703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:54.772799 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.772833 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 09:07:54.772895 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 09:07:54.772988 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":304,"thread_start_us":143,"threads_started":1}
I20260620 09:07:54.773849 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac79d8383e49484c8cc4111841da1040" candidate_uuid: "814c15fa513c4b7ba3a33922726009a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" is_pre_election: true
I20260620 09:07:54.773825 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac79d8383e49484c8cc4111841da1040" candidate_uuid: "814c15fa513c4b7ba3a33922726009a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb2001f29684f74b61565f85c61f1ec" is_pre_election: true
I20260620 09:07:54.774165 14981 raft_consensus.cc:2454] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c15fa513c4b7ba3a33922726009a0 in term 0.
I20260620 09:07:54.774165 14843 raft_consensus.cc:2454] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c15fa513c4b7ba3a33922726009a0 in term 0.
I20260620 09:07:54.774664 15048 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0, bbb2001f29684f74b61565f85c61f1ec; no voters:
I20260620 09:07:54.774966 15176 raft_consensus.cc:2781] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:07:54.775067 15176 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:07:54.775125 15176 raft_consensus.cc:3037] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:54.776252 15176 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.776670 15176 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [CANDIDATE]: Term 1 election: Requested vote from peers 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333), bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199)
I20260620 09:07:54.777122 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac79d8383e49484c8cc4111841da1040" candidate_uuid: "814c15fa513c4b7ba3a33922726009a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb2001f29684f74b61565f85c61f1ec"
I20260620 09:07:54.777128 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac79d8383e49484c8cc4111841da1040" candidate_uuid: "814c15fa513c4b7ba3a33922726009a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
I20260620 09:07:54.777290 14843 raft_consensus.cc:3037] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:54.777292 14981 raft_consensus.cc:3037] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:07:54.778556 14981 raft_consensus.cc:2454] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c15fa513c4b7ba3a33922726009a0 in term 1.
I20260620 09:07:54.778564 14843 raft_consensus.cc:2454] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c15fa513c4b7ba3a33922726009a0 in term 1.
I20260620 09:07:54.778857 15048 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0; no voters:
I20260620 09:07:54.779130 15176 raft_consensus.cc:2781] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:07:54.779364 15176 raft_consensus.cc:686] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 1 LEADER]: Becoming Leader. State: Replica: 814c15fa513c4b7ba3a33922726009a0, State: Running, Role: LEADER
I20260620 09:07:54.779755 15176 consensus_queue.cc:237] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:54.783283 14701 catalog_manager.cc:5495] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "814c15fa513c4b7ba3a33922726009a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } health_report { overall_health: UNKNOWN } } }
I20260620 09:07:54.808663 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.804979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.3:33355)
0620 09:07:54.805140 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:54.805144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:54.805161 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:07:54.805291 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:54.805294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:54.805361 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:54.805482 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:07:54.805740 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.806264 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.807105 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:54.807340 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:54.807448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:54.807795 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:07:54.808242 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:07:54.808386 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:54.808426 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 09:07:54.808485 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 09:07:54.814772 14981 raft_consensus.cc:1261] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Refusing update from remote peer 814c15fa513c4b7ba3a33922726009a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:54.814770 14843 raft_consensus.cc:1261] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Refusing update from remote peer 814c15fa513c4b7ba3a33922726009a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:07:54.815512 15176 consensus_queue.cc:1035] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:54.815753 15180 consensus_queue.cc:1035] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:07:54.822337 15183 mvcc.cc:205] Tried to move back new op lower bound from 7298852760833740800 to 7298852760700882944. Current Snapshot: MvccSnapshot[applied={T|T < 7298852760833740800}]
I20260620 09:07:54.822579 15184 mvcc.cc:205] Tried to move back new op lower bound from 7298852760833740800 to 7298852760700882944. Current Snapshot: MvccSnapshot[applied={T|T < 7298852760833740800}]
I20260620 09:07:54.823704 15185 mvcc.cc:205] Tried to move back new op lower bound from 7298852760833740800 to 7298852760700882944. Current Snapshot: MvccSnapshot[applied={T|T < 7298852760833740800}]
I20260620 09:07:54.833003 14701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:54.833276 14701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:07:54.835772 14701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Sending DeleteTablet for 3 replicas of tablet ac79d8383e49484c8cc4111841da1040
I20260620 09:07:54.836555 14823 tablet_service.cc:1514] Processing DeleteTablet for tablet ac79d8383e49484c8cc4111841da1040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55476
I20260620 09:07:54.836618 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet ac79d8383e49484c8cc4111841da1040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38424
I20260620 09:07:54.836699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14739
I20260620 09:07:54.837500 15194 tablet_replica.cc:330] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: stopping tablet replica
I20260620 09:07:54.837529 15093 tablet_service.cc:1514] Processing DeleteTablet for tablet ac79d8383e49484c8cc4111841da1040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:07:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40652
I20260620 09:07:54.837925 15194 raft_consensus.cc:2229] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:07:54.838033 15195 tablet_replica.cc:330] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: stopping tablet replica
I20260620 09:07:54.838302 15195 raft_consensus.cc:2229] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:07:54.838325 15194 raft_consensus.cc:2258] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:54.838722 15195 raft_consensus.cc:2258] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:07:54.840082 15194 ts_tablet_manager.cc:1905] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:54.840082 15195 ts_tablet_manager.cc:1905] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:07:54.842926 15195 ts_tablet_manager.cc:1918] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:54.842926 15194 ts_tablet_manager.cc:1918] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:07:54.843035 15195 log.cc:1196] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/ac79d8383e49484c8cc4111841da1040
I20260620 09:07:54.843107 15194 log.cc:1196] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/ac79d8383e49484c8cc4111841da1040
I20260620 09:07:54.843422 15195 ts_tablet_manager.cc:1939] T ac79d8383e49484c8cc4111841da1040 P 814c15fa513c4b7ba3a33922726009a0: Deleting consensus metadata
I20260620 09:07:54.843502 15194 ts_tablet_manager.cc:1939] T ac79d8383e49484c8cc4111841da1040 P 50aa17f1d76f49a9a83d18dcce508c9d: Deleting consensus metadata
I20260620 09:07:54.844710 14685 catalog_manager.cc:4841] TS 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): tablet ac79d8383e49484c8cc4111841da1040 (table test-table [id=f63283c398c6454388d317aeea56f3cd]) successfully deleted
I20260620 09:07:54.844712 14687 catalog_manager.cc:4841] TS 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): tablet ac79d8383e49484c8cc4111841da1040 (table test-table [id=f63283c398c6454388d317aeea56f3cd]) successfully deleted
W20260620 09:07:54.846544 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38199 (error 108)
I20260620 09:07:54.847354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14893
I20260620 09:07:54.849922 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:54.849621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38199 (local address 127.0.0.1:55490)
0620 09:07:54.849766 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:54.849844 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 09:07:54.850140 14685 catalog_manager.cc:4584] TS bbb2001f29684f74b61565f85c61f1ec (127.0.98.1:38199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac79d8383e49484c8cc4111841da1040: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38199: connect: Connection refused (error 111)
I20260620 09:07:54.854929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15030
I20260620 09:07:54.860028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 14668
I20260620 09:07:54.866359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--webserver_interface=127.0.98.62
--webserver_port=43883
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:07:54.980003 15197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:54.980322 15197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:54.980407 15197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:54.984716 15197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:07:54.984788 15197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:54.984808 15197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:54.984825 15197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:07:54.984841 15197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:07:54.990446 15197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:54.991577 15197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:54.992604 15197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:55.001438 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.001433 15203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.001487 15197 server_base.cc:1034] running on GCE node
W20260620 09:07:55.001433 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.002345 15197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:55.003587 15197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:55.004805 15197 hybrid_clock.cc:648] HybridClock initialized: now 1781946475004789 us; error 30 us; skew 500 ppm
Jun 20 09:07:55 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:55.007787 15197 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:07:55.008772 15197 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260620 09:07:55.009415 15197 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:55.009483 15197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:55.013687 15197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 09:07:55.016360 15212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.017768 15197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:55.017920 15197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "90915675417c4424b04b16bc5d70d638"
format_stamp: "Formatted at 2026-06-20 09:07:52 on dist-test-slave-92t4"
server_key: "4aa8effe06fe17fa009ae915b82e8289"
server_key_iv: "0d887f4440415a2e1e2aa76e1918d114"
server_key_version: "encryptionkey@0"
I20260620 09:07:55.018523 15197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:55.035236 15197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:55.038931 15197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:55.039177 15197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:55.047068 15197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33539
I20260620 09:07:55.047374 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33539 every 8 connection(s)
I20260620 09:07:55.048276 15197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:07:55.052793 15265 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:07:55.053239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15197
I20260620 09:07:55.053717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38199
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=46865
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:55.055321 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap starting.
I20260620 09:07:55.073830 15265 log.cc:824] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:55.085659 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:07:55.086126 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638: Bootstrap complete.
I20260620 09:07:55.090497 15265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:55.090902 15265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Initialized, Role: FOLLOWER
I20260620 09:07:55.091650 15265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:55.091794 15265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:07:55.091988 15265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:07:55.092115 15265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:07:55.094313 15265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:55.094691 15265 leader_election.cc:302] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90915675417c4424b04b16bc5d70d638; no voters:
I20260620 09:07:55.095069 15265 leader_election.cc:288] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:07:55.095163 15269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:07:55.095357 15269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [term 3 LEADER]: Becoming Leader. State: Replica: 90915675417c4424b04b16bc5d70d638, State: Running, Role: LEADER
I20260620 09:07:55.095683 15269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } }
I20260620 09:07:55.096386 15265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:07:55.096704 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90915675417c4424b04b16bc5d70d638. Latest consensus state: current_term: 3 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:55.096835 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:55.097059 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "90915675417c4424b04b16bc5d70d638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90915675417c4424b04b16bc5d70d638" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33539 } } }
I20260620 09:07:55.097159 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90915675417c4424b04b16bc5d70d638 [sys.catalog]: This master's current role is: LEADER
I20260620 09:07:55.097220 15273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:07:55.102880 15273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:07:55.103624 15273 catalog_manager.cc:1259] Loaded cluster ID: 59900dc06f314e09aef34041c81af6d3
I20260620 09:07:55.103690 15273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 09:07:55.104753 15273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f07750a79ad google::LogMessage::Fail() at ??:0
@ 0x7f07750aa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f07750a74de google::LogMessage::Flush() at ??:0
@ 0x7f07750a8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f077f9ecfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f077f9edd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f077f9ea579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f077fa2e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f07772f93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f07772f9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f07772fb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f07772ea512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f07789326db start_thread at ??:0
@ 0x7f0773ba561f clone at ??:0
W20260620 09:07:55.186769 15267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:55.187096 15267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:55.187208 15267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:55.191636 15267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:55.191732 15267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:55.191836 15267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:07:55.197495 15267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46865
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:55.198693 15267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:55.199893 15267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:55.209971 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.210009 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.209976 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.210292 15267 server_base.cc:1034] running on GCE node
I20260620 09:07:55.210757 15267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:55.211614 15267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:55.212822 15267 hybrid_clock.cc:648] HybridClock initialized: now 1781946475212764 us; error 66 us; skew 500 ppm
Jun 20 09:07:55 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:55.216141 15267 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:07:55.217160 15267 webserver.cc:460] Webserver started at http://127.0.98.1:46865/ using document root <none> and password file <none>
I20260620 09:07:55.217744 15267 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:55.217805 15267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:55.222021 15267 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:07:55.224435 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.225981 15267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:07:55.226142 15267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "bbb2001f29684f74b61565f85c61f1ec"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "6b54b67a334b1cdca8d0d5afd3c612c9"
server_key_iv: "f89b9ff338aa55823a3d435ea8555270"
server_key_version: "encryptionkey@0"
I20260620 09:07:55.226773 15267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:55.259316 15267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:55.263391 15267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:55.263631 15267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:55.264343 15267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:55.265705 15323 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:07:55.271534 15267 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:07:55.271661 15267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 09:07:55.271762 15267 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:07:55.273766 15267 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:07:55.273859 15267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 09:07:55.274160 15323 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Bootstrap starting.
I20260620 09:07:55.285061 15267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38199
I20260620 09:07:55.285246 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38199 every 8 connection(s)
I20260620 09:07:55.286235 15267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:07:55.291632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15267
I20260620 09:07:55.292128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39333
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=34019
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:55.296463 15323 log.cc:824] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Log is configured to *not* fsync() on all Append() calls
I20260620 09:07:55.306893 15323 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:07:55.307361 15323 tablet_bootstrap.cc:492] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Bootstrap complete.
I20260620 09:07:55.308889 15323 ts_tablet_manager.cc:1397] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20260620 09:07:55.312374 15323 raft_consensus.cc:348] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:55.312973 15323 raft_consensus.cc:729] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb2001f29684f74b61565f85c61f1ec, State: Initialized, Role: FOLLOWER
I20260620 09:07:55.313561 15323 consensus_queue.cc:260] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:55.314723 15323 ts_tablet_manager.cc:1428] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
W20260620 09:07:55.353561 15433 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:07:55.288373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:35207)
0620 09:07:55.288885 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.288944 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:55.289836 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:55.352660 (+ 62824us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":153,"threads_started":1}
W20260620 09:07:55.354090 15431 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33539: Client connection negotiation failed: client connection to 127.0.98.62:33539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:07:55.354909 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.354626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:54655)
0620 09:07:55.354760 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.354825 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 09:07:55.355654 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.355466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:47549)
0620 09:07:55.355569 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.355610 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:07:55.355934 15431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:07:55.416052 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:55.416420 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:55.416514 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:55.420694 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:55.420822 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:55.420931 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:07:55.426653 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34019
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:55.427804 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:55.428900 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:55.439182 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.439188 15441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.439219 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.439601 15434 server_base.cc:1034] running on GCE node
I20260620 09:07:55.440045 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:55.440688 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:55.441875 15434 hybrid_clock.cc:648] HybridClock initialized: now 1781946475441857 us; error 28 us; skew 500 ppm
Jun 20 09:07:55 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:55.444821 15434 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:07:55.445863 15434 webserver.cc:460] Webserver started at http://127.0.98.2:34019/ using document root <none> and password file <none>
I20260620 09:07:55.446415 15434 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:55.446477 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:55.450500 15434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:07:55.452762 15451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.454293 15434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:07:55.454449 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "50aa17f1d76f49a9a83d18dcce508c9d"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "78de4fb39885efdecc72300fb4df79e9"
server_key_iv: "b4ee2d7d9d433c1764a17bb91ded3f1c"
server_key_version: "encryptionkey@0"
I20260620 09:07:55.455073 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:55.470214 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:55.474040 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:55.474268 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:55.474933 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:55.475816 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:55.475864 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.475939 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:55.475980 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.486325 15434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39333
I20260620 09:07:55.486348 15564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39333 every 8 connection(s)
I20260620 09:07:55.487515 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:07:55.491765 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.489906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:34975)
0620 09:07:55.490393 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.490899 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260620 09:07:55.491902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15434
W20260620 09:07:55.492261 15565 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33539: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
I20260620 09:07:55.492365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33355
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33539
--webserver_port=38919
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:07:55.493089 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.492730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:46939)
0620 09:07:55.492882 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.492951 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 09:07:55.493847 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.493566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:45503)
0620 09:07:55.493704 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.493768 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 09:07:55.494055 15565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:07:55.609506 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:07:55.609820 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:07:55.609866 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:07:55.614135 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:07:55.614207 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:07:55.614270 15568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:07:55.620357 15568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38919
--tserver_master_addrs=127.0.98.62:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:07:55.621556 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:07:55.622582 15568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:07:55.632838 15574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:07:55.632813 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.633033 15568 server_base.cc:1034] running on GCE node
W20260620 09:07:55.632822 15576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:07:55.633576 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:07:55.634305 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:07:55.635486 15568 hybrid_clock.cc:648] HybridClock initialized: now 1781946475635472 us; error 26 us; skew 500 ppm
Jun 20 09:07:55 dist-test-slave-92t4 krb5kdc[14120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:07:55.638470 15568 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:07:55.639441 15568 webserver.cc:460] Webserver started at http://127.0.98.3:38919/ using document root <none> and password file <none>
I20260620 09:07:55.640004 15568 fs_manager.cc:359] Metadata directory not provided
I20260620 09:07:55.640066 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:07:55.644320 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:07:55.646678 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.648135 15568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:07:55.648255 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "814c15fa513c4b7ba3a33922726009a0"
format_stamp: "Formatted at 2026-06-20 09:07:53 on dist-test-slave-92t4"
server_key: "d8805effdaf1922914662a0b11d4c886"
server_key_iv: "553457977d341419e4ed667f2c07e690"
server_key_version: "encryptionkey@0"
I20260620 09:07:55.648945 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:07:55.673223 15568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:07:55.676839 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:07:55.677011 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:07:55.677623 15568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:07:55.678514 15568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:07:55.678561 15568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.678602 15568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:07:55.678618 15568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:07:55.688470 15568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33355
I20260620 09:07:55.688506 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33355 every 8 connection(s)
I20260620 09:07:55.689711 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:07:55.693897 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.692049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:54893)
0620 09:07:55.692553 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.693129 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
W20260620 09:07:55.694391 15697 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33539: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
I20260620 09:07:55.695050 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.694805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:53881)
0620 09:07:55.694947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.694996 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 09:07:55.695633 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:55.695456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:40583)
0620 09:07:55.695547 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:55.695587 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:07:55.695811 15697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:07:55.699980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15568
I20260620 09:07:56.357004 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.356522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:60363)
0620 09:07:56.356835 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:56.356900 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260620 09:07:56.495157 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.494689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:43741)
0620 09:07:56.494970 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:56.495047 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260620 09:07:56.696904 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.696421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:45299)
0620 09:07:56.696717 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:56.696770 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260620 09:07:56.965639 15704 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:56.965885 15704 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:56.967093 15704 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355), 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
Jun 20 09:07:56 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 09:07:56 dist-test-slave-92t4 krb5kdc[14120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 09:07:56.981386 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.967854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53621 (local address 127.0.98.2:39333)
0620 09:07:56.968149 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:56.968174 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:56.968210 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:07:56.968990 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:56.968996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:56.969451 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:56.969801 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:56.970918 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.972176 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.973419 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.973869 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.974038 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:56.976942 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:56.976947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:56.976958 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:56.976987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:56.978951 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979422 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:56.979427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:56.979432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:56.979554 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979849 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:56.979853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:56.979854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:56.980225 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:56.980365 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:56.980874 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 09:07:56.981179 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260620 09:07:56.981380 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.967407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39333 (local address 127.0.98.1:53621)
0620 09:07:56.967950 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:56.967965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:56.968102 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:56.970314 (+ 2212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:56.970317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:56.970493 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:56.970731 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:56.970746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.972338 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.972341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:56.973267 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:56.973276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.973994 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.973997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:56.974205 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:56.974918 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:56.974934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:56.976757 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:56.979100 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:56.979105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:56.979307 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:56.979662 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:56.979667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:56.979753 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:56.980335 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:56.980338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:56.980576 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 09:07:56.980767 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 09:07:56.981179 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":297,"spinlock_wait_cycles":5248,"thread_start_us":70,"threads_started":1}
I20260620 09:07:56.981413 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.967557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46991 (local address 127.0.98.3:33355)
0620 09:07:56.967907 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 09:07:56.967938 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 09:07:56.967995 (+ 57us) server_negotiation.cc:407] Connection header received
0620 09:07:56.968934 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:07:56.968944 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:07:56.969451 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:07:56.969801 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:07:56.970931 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.972154 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.973388 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.973877 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.974054 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:56.976814 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:07:56.976819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:07:56.976831 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:07:56.976866 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:07:56.978693 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979284 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:56.979290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:56.979295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:56.979421 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979756 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:07:56.979761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:07:56.979762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:07:56.980178 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:07:56.980325 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:07:56.980917 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 09:07:56.981217 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260620 09:07:56.981484 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:56.967407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33355 (local address 127.0.98.1:46991)
0620 09:07:56.967747 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:56.967766 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:07:56.967882 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:07:56.969850 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:07:56.969855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:07:56.970251 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:07:56.970731 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:56.970747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.972295 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.972299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:56.973261 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:07:56.973270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:07:56.974027 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:07:56.974030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:07:56.974260 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:07:56.974974 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:07:56.974986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:07:56.976571 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:07:56.978821 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:56.978873 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:56.978890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:56.979163 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:56.979518 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:07:56.979523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:07:56.979525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:07:56.979662 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:07:56.980299 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:07:56.980306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:07:56.980576 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 09:07:56.980783 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 09:07:56.981264 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":281,"spinlock_wait_cycles":40192,"thread_start_us":89,"threads_started":1}
W20260620 09:07:56.984124 15317 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
W20260620 09:07:56.984289 15319 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
I20260620 09:07:56.984395 15317 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb2001f29684f74b61565f85c61f1ec; no voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0
I20260620 09:07:56.984720 15704 raft_consensus.cc:2726] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:07:57.358376 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:57.358036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:38937)
0620 09:07:57.358201 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:57.358284 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 09:07:57.496433 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:57.495980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:54087)
0620 09:07:57.496251 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:57.496314 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260620 09:07:57.698233 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:57.697794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:50547)
0620 09:07:57.698072 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:57.698137 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260620 09:07:58.359776 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:58.359330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:56159)
0620 09:07:58.359609 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:58.359675 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260620 09:07:58.497882 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:58.497422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:34381)
0620 09:07:58.497701 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:58.497771 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260620 09:07:58.699457 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:58.698998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:40029)
0620 09:07:58.699281 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:58.699337 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260620 09:07:58.885941 15714 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:07:58.886114 15714 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:07:58.886641 15714 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355), 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
W20260620 09:07:58.887537 15317 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
W20260620 09:07:58.887627 15319 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
I20260620 09:07:58.887745 15319 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb2001f29684f74b61565f85c61f1ec; no voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0
I20260620 09:07:58.887955 15714 raft_consensus.cc:2726] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:07:59.361238 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:59.360687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:51143)
0620 09:07:59.360981 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:59.361117 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260620 09:07:59.499253 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:59.498772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:36999)
0620 09:07:59.499093 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:59.499153 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260620 09:07:59.700709 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:07:59.700289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:47783)
0620 09:07:59.700551 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 09:07:59.700612 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260620 09:08:00.362658 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:00.362184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:40669)
0620 09:08:00.362505 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:00.362559 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260620 09:08:00.500571 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:00.500150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:52667)
0620 09:08:00.500411 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:00.500471 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260620 09:08:00.702144 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:00.701693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:60317)
0620 09:08:00.701968 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:00.702034 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260620 09:08:01.363957 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:01.363466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:41587)
0620 09:08:01.363766 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:01.363840 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260620 09:08:01.502084 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:01.501601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:60167)
0620 09:08:01.501915 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:01.501984 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
I20260620 09:08:01.703346 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:01.702928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:37045)
0620 09:08:01.703191 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:01.703244 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260620 09:08:01.734202 15724 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:01.734390 15724 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:08:01.734870 15724 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355), 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
W20260620 09:08:01.735682 15317 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
W20260620 09:08:01.735895 15319 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
I20260620 09:08:01.735972 15319 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb2001f29684f74b61565f85c61f1ec; no voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0
I20260620 09:08:01.736173 15724 raft_consensus.cc:2726] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:08:02.365327 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:02.364873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:40767)
0620 09:08:02.365173 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:02.365227 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
I20260620 09:08:02.503472 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:02.502924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:52883)
0620 09:08:02.503288 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:02.503374 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":75,"threads_started":1}
I20260620 09:08:02.704725 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:02.704207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:40203)
0620 09:08:02.704559 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:02.704625 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260620 09:08:03.366812 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:03.366333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:35285)
0620 09:08:03.366624 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:03.366685 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260620 09:08:03.504730 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:03.504324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:37229)
0620 09:08:03.504571 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:03.504633 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260620 09:08:03.706161 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:03.705687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:51325)
0620 09:08:03.705992 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:03.706059 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260620 09:08:04.285346 15731 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:04.285486 15731 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:08:04.285993 15731 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355), 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
W20260620 09:08:04.286854 15319 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
W20260620 09:08:04.286949 15317 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
I20260620 09:08:04.287042 15317 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb2001f29684f74b61565f85c61f1ec; no voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0
I20260620 09:08:04.287189 15731 raft_consensus.cc:2726] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:08:04.368124 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:04.367709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:37829)
0620 09:08:04.367972 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:04.368028 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260620 09:08:04.506139 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:04.505620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:44523)
0620 09:08:04.505956 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:04.506025 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260620 09:08:04.707631 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:04.707130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:36475)
0620 09:08:04.707454 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:04.707512 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260620 09:08:05.369516 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:05.369027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:38959)
0620 09:08:05.369360 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:05.369418 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":72,"threads_started":1}
I20260620 09:08:05.507474 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:05.507040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:43217)
0620 09:08:05.507295 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:05.507374 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260620 09:08:05.708846 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:05.708438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:35645)
0620 09:08:05.708689 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:05.708745 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260620 09:08:06.370790 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:06.370358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:46297)
0620 09:08:06.370622 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:06.370692 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260620 09:08:06.508906 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:06.508397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:37981)
0620 09:08:06.508703 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:06.508800 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260620 09:08:06.710065 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:06.709633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:36697)
0620 09:08:06.709910 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:06.709970 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260620 09:08:07.372215 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:07.371730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:42209)
0620 09:08:07.372048 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:07.372115 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260620 09:08:07.510140 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:07.509716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:56353)
0620 09:08:07.509982 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:07.510042 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260620 09:08:07.711524 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:07.711023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:38017)
0620 09:08:07.711359 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:07.711426 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":89,"threads_started":1}
I20260620 09:08:08.298604 15744 raft_consensus.cc:482] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:08.298847 15744 raft_consensus.cc:504] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814c15fa513c4b7ba3a33922726009a0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33355 } } peers { permanent_uuid: "50aa17f1d76f49a9a83d18dcce508c9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39333 } } peers { permanent_uuid: "bbb2001f29684f74b61565f85c61f1ec" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38199 } }
I20260620 09:08:08.299682 15744 leader_election.cc:288] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355), 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333)
W20260620 09:08:08.300092 15319 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50aa17f1d76f49a9a83d18dcce508c9d (127.0.98.2:39333): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
W20260620 09:08:08.300266 15317 leader_election.cc:341] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 814c15fa513c4b7ba3a33922726009a0 (127.0.98.3:33355): Not found: Tablet not found: ac79d8383e49484c8cc4111841da1040
I20260620 09:08:08.300364 15317 leader_election.cc:302] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb2001f29684f74b61565f85c61f1ec; no voters: 50aa17f1d76f49a9a83d18dcce508c9d, 814c15fa513c4b7ba3a33922726009a0
I20260620 09:08:08.300530 15744 raft_consensus.cc:2726] T ac79d8383e49484c8cc4111841da1040 P bbb2001f29684f74b61565f85c61f1ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:08:08.373798 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:08.373316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:37533)
0620 09:08:08.373625 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:08.373683 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260620 09:08:08.511543 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:08.511099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:50997)
0620 09:08:08.511373 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:08.511445 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260620 09:08:08.712996 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:08.712500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:43683)
0620 09:08:08.712805 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:08.712866 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260620 09:08:09.374971 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:09.374507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:45033)
0620 09:08:09.374808 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:09.374864 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20260620 09:08:09.513003 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:09.512515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:46381)
0620 09:08:09.512809 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:09.512873 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260620 09:08:09.714520 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:09.714049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.3:48949)
0620 09:08:09.714315 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:09.714374 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260620 09:08:10.376500 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:10.376060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.1:35027)
0620 09:08:10.376344 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:10.376405 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260620 09:08:10.514542 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:10.513971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33539 (local address 127.0.98.2:39971)
0620 09:08:10.514309 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:10.514410 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":90,"threads_started":1}
I20260620 09:08:10.701803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15267
I20260620 09:08:10.707955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15434
I20260620 09:08:10.713306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15568
2026-06-20T09:08:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19308 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:08:10 dist-test-slave-92t4 krb5kdc[15755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:08:10 dist-test-slave-92t4 krb5kdc[15755](info): set up 2 sockets
Jun 20 09:08:10 dist-test-slave-92t4 krb5kdc[15755](info): commencing operation
krb5kdc: starting...
W20260620 09:08:11.749449 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:08:11 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:08:11.872071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:11.987526 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:11.987845 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:11.987924 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:11.992010 15768 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:08:11.992111 15768 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:08:11.992137 15768 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:08:11.992161 15768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:08:11.992209 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:11.992393 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:11.992472 15768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:08:11.992518 15768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:08:11.998358 15768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:11.999552 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:12.000663 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:12.009877 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.009877 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.009915 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:12.009938 15768 server_base.cc:1034] running on GCE node
I20260620 09:08:12.010850 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:12.012058 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:12.013233 15768 hybrid_clock.cc:648] HybridClock initialized: now 1781946492013215 us; error 35 us; skew 500 ppm
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:12.016125 15768 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:08:12.016999 15768 webserver.cc:460] Webserver started at http://127.0.98.62:38583/ using document root <none> and password file <none>
I20260620 09:08:12.017599 15768 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:12.017661 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:12.017889 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:08:12.019958 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "617b926feb6b463498573d88c1af5b9b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "8c66c4d462781bb3d34753a5ae972e4e"
server_key_iv: "fc6586362395fdfd581bcd93d3967c04"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.020478 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "617b926feb6b463498573d88c1af5b9b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "8c66c4d462781bb3d34753a5ae972e4e"
server_key_iv: "fc6586362395fdfd581bcd93d3967c04"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.024539 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:08:12.027271 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.028687 15768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:08:12.028861 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "617b926feb6b463498573d88c1af5b9b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "8c66c4d462781bb3d34753a5ae972e4e"
server_key_iv: "fc6586362395fdfd581bcd93d3967c04"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.028990 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:12.039963 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:12.043390 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:12.043606 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:12.051112 15768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33061
I20260620 09:08:12.051417 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33061 every 8 connection(s)
I20260620 09:08:12.052171 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:08:12.055081 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:12.058698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15768
I20260620 09:08:12.058808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:08:12.059108 392 external_mini_cluster.cc:1428] Setting key a64ceefe48523199f96d798f84bd0464
I20260620 09:08:12.061462 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap starting.
I20260620 09:08:12.064172 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:12.065021 15836 log.cc:824] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:12.067498 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: No bootstrap required, opened a new log
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:08:12.071271 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:12.071542 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:12.071626 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Initialized, Role: FOLLOWER
I20260620 09:08:12.072122 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:12.072286 15836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:08:12.072374 15836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:08:12.072484 15836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:12.073758 15836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:12.074126 15836 leader_election.cc:302] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617b926feb6b463498573d88c1af5b9b; no voters:
I20260620 09:08:12.074483 15836 leader_election.cc:288] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:08:12.074635 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:08:12.074867 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 LEADER]: Becoming Leader. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Running, Role: LEADER
I20260620 09:08:12.075207 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.060667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.0.98.62:33061)
0620 09:08:12.061139 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.061155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.061193 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:08:12.061937 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.061954 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.062315 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.062612 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:12.063601 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.064731 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.065902 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.066444 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.066573 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.069509 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:12.069515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:12.069531 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:12.069560 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:12.071958 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.072440 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.072446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.072452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.072603 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.072896 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.072899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.072901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.073452 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:12.073669 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.073882 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.074195 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260620 09:08:12.075227 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:12.075803 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:08:12.076988 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:12.077128 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
I20260620 09:08:12.077579 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617b926feb6b463498573d88c1af5b9b. Latest consensus state: current_term: 1 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:12.077718 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
W20260620 09:08:12.080380 15856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:08:12.080484 15856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:08:12.080549 15850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:08:12.081552 15850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:08:12.087670 15850 catalog_manager.cc:1347] Generated new cluster ID: 5b66b21584d24952886973dab2a30939
I20260620 09:08:12.087793 15850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:08:12.098820 15850 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:08:12.099813 15850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:08:12.107585 15850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Generated new TSK 0
I20260620 09:08:12.108351 15850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:08:12.151810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:12.277318 15862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:12.277583 15862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:12.277666 15862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:12.281730 15862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:12.281793 15862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:12.281903 15862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:08:12.287237 15862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:12.288350 15862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:12.289461 15862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:12.299067 15868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.299078 15870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.299089 15867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:12.299319 15862 server_base.cc:1034] running on GCE node
I20260620 09:08:12.299803 15862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:12.300444 15862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:12.301601 15862 hybrid_clock.cc:648] HybridClock initialized: now 1781946492301586 us; error 28 us; skew 500 ppm
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:12.304271 15862 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:08:12.305213 15862 webserver.cc:460] Webserver started at http://127.0.98.1:40225/ using document root <none> and password file <none>
I20260620 09:08:12.305867 15862 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:12.305963 15862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:12.306206 15862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:08:12.308360 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "4f7c282c646e49582ada06c8cb060803"
server_key_iv: "55c6e8e36865b5d4716a2bdda55697bc"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.308947 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "4f7c282c646e49582ada06c8cb060803"
server_key_iv: "55c6e8e36865b5d4716a2bdda55697bc"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.312767 15862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:08:12.315284 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.316459 15862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:08:12.316592 15862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "4f7c282c646e49582ada06c8cb060803"
server_key_iv: "55c6e8e36865b5d4716a2bdda55697bc"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.316717 15862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:12.341992 15862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:12.344861 15862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:12.345156 15862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:12.345795 15862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:12.346647 15862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:12.346722 15862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.346839 15862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:12.346890 15862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.355612 15862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41639
I20260620 09:08:12.355629 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41639 every 8 connection(s)
I20260620 09:08:12.356541 15862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:08:12.359462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15862
I20260620 09:08:12.359591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:08:12.359869 392 external_mini_cluster.cc:1428] Setting key 655602064e44637200f02ce2e12c2229
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:08:12.370705 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.358383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59727 (local address 127.0.98.62:33061)
0620 09:08:12.358548 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.358553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.359401 (+ 848us) server_negotiation.cc:407] Connection header received
0620 09:08:12.360829 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.360833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.360884 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.360993 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:12.362215 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.362733 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.363880 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.364116 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.364224 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.367065 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:12.367068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:12.367070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:12.367094 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:12.368597 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.369155 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.369158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.369160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.369254 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.369673 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.369676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.369677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.369878 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:12.369963 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.370468 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.370578 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:08:12.371729 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.358782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:59727)
0620 09:08:12.359235 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.359291 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.360613 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.361219 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.361226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.361595 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:12.362069 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.362082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.362867 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.362871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.363754 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.363763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.364292 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.364297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.364461 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.365123 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:12.365140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:12.366886 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:12.368721 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.368728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.368754 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.369021 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.369356 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.369359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.369361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.369557 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.369976 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:12.369981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:12.370241 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.370406 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.370778 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260620 09:08:12.373082 15991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:12.373373 15991 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:12.374012 15991 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:12.375627 15800 ts_manager.cc:194] Registered new tserver with Master: 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639)
I20260620 09:08:12.376873 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:08:12.388132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:12.500873 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:12.501176 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:12.501222 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:12.505352 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:12.505434 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:12.505496 15996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:08:12.510805 15996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:12.511860 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:12.512859 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:12.523013 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.522979 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.522995 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:12.523281 15996 server_base.cc:1034] running on GCE node
I20260620 09:08:12.523717 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:12.524430 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:12.525583 15996 hybrid_clock.cc:648] HybridClock initialized: now 1781946492525547 us; error 51 us; skew 500 ppm
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:12.528352 15996 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:08:12.529428 15996 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260620 09:08:12.529987 15996 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:12.530045 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:12.530212 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:08:12.532249 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c10a16140024fe091049b2d2b672e3b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "68b0ababb362bfb51f3827f9e397a407"
server_key_iv: "b42184a17b8c73477cd61dc9ac4db644"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.532773 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c10a16140024fe091049b2d2b672e3b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "68b0ababb362bfb51f3827f9e397a407"
server_key_iv: "b42184a17b8c73477cd61dc9ac4db644"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.536950 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:08:12.539460 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.540728 15996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:08:12.540858 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "9c10a16140024fe091049b2d2b672e3b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "68b0ababb362bfb51f3827f9e397a407"
server_key_iv: "b42184a17b8c73477cd61dc9ac4db644"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.540968 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:12.557459 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:12.560375 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:12.560587 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:12.561278 15996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:12.562117 15996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:12.562172 15996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.562244 15996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:12.562283 15996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.571231 15996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260620 09:08:12.571252 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260620 09:08:12.572252 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:08:12.574853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 15996
I20260620 09:08:12.574980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:08:12.575304 392 external_mini_cluster.cc:1428] Setting key 429a81819948959f35120dd3c9bd8e2d
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:08:12.587225 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.574218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38441 (local address 127.0.98.62:33061)
0620 09:08:12.574417 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.574422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.575527 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 09:08:12.577212 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.577215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.577269 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.577378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:12.578939 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.579465 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.580651 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.580884 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.581000 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.583471 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:12.583474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:12.583477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:12.583500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:12.585004 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.585585 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.585600 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.585602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.585684 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.586034 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.586039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.586041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.586235 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:12.586370 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.586919 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.587051 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 09:08:12.588312 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.574700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:38441)
0620 09:08:12.575282 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.575336 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.576954 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.577571 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.577579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.578099 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:12.578765 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.578786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.579592 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.579595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.580520 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.580529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.581062 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.581067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.581244 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.581871 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:12.581887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:12.583294 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:12.585184 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.585191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.585206 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.585473 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.585780 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.585784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.585786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.585935 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.586368 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:12.586375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:12.586637 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.586861 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.587210 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":151,"threads_started":1}
I20260620 09:08:12.589670 16125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:12.589932 16125 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:12.590444 16125 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:12.591563 15800 ts_manager.cc:194] Registered new tserver with Master: 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529)
I20260620 09:08:12.592149 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:08:12.604079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:12.716558 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:12.716847 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:12.716934 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:12.721113 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:12.721175 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:12.721271 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:08:12.726773 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:12.727947 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:12.729096 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:12.739365 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.739382 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:12.739365 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:12.739928 16130 server_base.cc:1034] running on GCE node
I20260620 09:08:12.740378 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:12.741124 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:12.742311 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781946492742295 us; error 36 us; skew 500 ppm
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:12.745260 16130 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:08:12.746203 16130 webserver.cc:460] Webserver started at http://127.0.98.3:36313/ using document root <none> and password file <none>
I20260620 09:08:12.746824 16130 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:12.746922 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:12.747159 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:08:12.749357 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf0874dbddb04923ad333adaade22785"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "63dc78f340de0a6475357ac02cbfbe61"
server_key_iv: "08232457c50cdd3ed24bb38af190dfa1"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.749969 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf0874dbddb04923ad333adaade22785"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "63dc78f340de0a6475357ac02cbfbe61"
server_key_iv: "08232457c50cdd3ed24bb38af190dfa1"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.753931 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:08:12.756459 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.757843 16130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 09:08:12.757997 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "cf0874dbddb04923ad333adaade22785"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "63dc78f340de0a6475357ac02cbfbe61"
server_key_iv: "08232457c50cdd3ed24bb38af190dfa1"
server_key_version: "encryptionkey@0"
I20260620 09:08:12.758124 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:12.778258 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:12.781749 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:12.781960 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:12.782575 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:12.783411 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:12.783479 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.783551 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:12.783601 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:12.792563 16130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33831
I20260620 09:08:12.792595 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33831 every 8 connection(s)
I20260620 09:08:12.793514 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:08:12.800987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16130
I20260620 09:08:12.801169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:08:12.801532 392 external_mini_cluster.cc:1428] Setting key 49f652d96af4204e5f1f50ea0695944b
Jun 20 09:08:12 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:08:12.807652 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.795362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45335 (local address 127.0.98.62:33061)
0620 09:08:12.795502 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.795505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.796384 (+ 879us) server_negotiation.cc:407] Connection header received
0620 09:08:12.797794 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.797797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.797855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.797966 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:12.799173 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.799707 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.800970 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.801338 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.801433 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.804019 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:12.804025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:12.804032 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:12.804058 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:12.805552 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.806084 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.806088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.806090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.806174 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.806521 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.806525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.806527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.806714 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:12.806833 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.807361 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.807490 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:08:12.808516 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.795703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:45335)
0620 09:08:12.796208 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.796269 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.797574 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.798147 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.798154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.798525 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:12.799020 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.799033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.799854 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.799857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.800825 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.800838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.801504 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.801509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.801727 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.802395 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:12.802411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:12.803856 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:12.805698 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.805705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.805720 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.805977 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.806272 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:12.806276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:12.806278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:12.806423 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:12.806832 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:12.806839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:12.807103 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.807321 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.807667 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260620 09:08:12.809741 16259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:12.810003 16259 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:12.810503 16259 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:12.811511 15800 ts_manager.cc:194] Registered new tserver with Master: cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
I20260620 09:08:12.812107 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45335
I20260620 09:08:12.815388 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:08:12.824558 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.817330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.0.98.62:33061)
0620 09:08:12.817477 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.817482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.817698 (+ 216us) server_negotiation.cc:407] Connection header received
0620 09:08:12.817818 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.817820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.817860 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.817925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:12.818579 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.819076 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.819857 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.820075 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.820192 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.821704 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:12.821707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:12.821709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:12.821731 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:12.823102 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.823487 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.823490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.823491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.823566 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:12.823798 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:12.823800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:12.823801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:12.824006 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:12.824106 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.824267 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.824363 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:08:12.827526 15800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:08:12.830586 15800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:08:12.846150 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.839504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.98.3:33831)
0620 09:08:12.841283 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.841294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.841324 (+ 30us) server_negotiation.cc:407] Connection header received
0620 09:08:12.841399 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.841403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.841578 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.841937 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:12.842062 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.842762 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.844413 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845346 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845518 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845655 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.845857 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.845976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":202,"threads_started":1}
I20260620 09:08:12.846184 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.839334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33831 (local address 127.0.0.1:40200)
0620 09:08:12.840132 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.840190 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.840332 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.841731 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.841736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.841756 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:12.841826 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.841830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.842903 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.842908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.844250 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.844260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845522 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.845658 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845708 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.845822 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.845979 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":118,"spinlock_wait_cycles":6656,"thread_start_us":80,"threads_started":1}
I20260620 09:08:12.846285 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.840298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.98.2:34529)
0620 09:08:12.840670 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.840683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.840977 (+ 294us) server_negotiation.cc:407] Connection header received
0620 09:08:12.841173 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.841177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.841386 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.841536 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:12.841734 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.843099 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.844658 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845358 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845484 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845654 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.846058 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.846156 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":136,"threads_started":1}
I20260620 09:08:12.846458 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.840113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.0.1:42924)
0620 09:08:12.840867 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.840881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.841007 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.841523 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.841526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.841542 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:12.841638 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.841648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.843249 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.843252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.844540 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.844547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845721 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.845822 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845853 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.845912 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.845979 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260620 09:08:12.847191 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.839581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.98.1:41639)
0620 09:08:12.840780 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.840798 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.840838 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:08:12.840910 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.840914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.841161 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.841331 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:12.841643 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.842923 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.844536 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845370 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845481 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845655 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.846937 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.847041 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":110,"threads_started":1}
I20260620 09:08:12.847440 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.839334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41639 (local address 127.0.0.1:38420)
0620 09:08:12.840131 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.840190 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.840332 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.841320 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.841326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.841346 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:12.841507 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.841514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.843078 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.843081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.844409 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.844420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.845744 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.845747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.845869 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.845891 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.845963 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.846031 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":533,"spinlock_wait_cycles":3328,"thread_start_us":89,"threads_started":1}
I20260620 09:08:12.849974 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e (DEFAULT_TABLE table=test-table [id=479e3aa099284b44adb96b648f1c42de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:12.849982 16059 tablet_service.cc:1467] Processing CreateTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e (DEFAULT_TABLE table=test-table [id=479e3aa099284b44adb96b648f1c42de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:12.849982 15925 tablet_service.cc:1467] Processing CreateTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e (DEFAULT_TABLE table=test-table [id=479e3aa099284b44adb96b648f1c42de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:12.851071 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9423dd57df0d4703b4ddb77e45777f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:12.851071 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9423dd57df0d4703b4ddb77e45777f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:12.851083 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9423dd57df0d4703b4ddb77e45777f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:12.857578 16276 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Bootstrap starting.
I20260620 09:08:12.858960 16278 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Bootstrap starting.
I20260620 09:08:12.859078 16277 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap starting.
I20260620 09:08:12.859895 16276 tablet_bootstrap.cc:654] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:12.860978 16276 log.cc:824] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:12.861222 16277 tablet_bootstrap.cc:654] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:12.861647 16278 tablet_bootstrap.cc:654] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:12.862553 16277 log.cc:824] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:12.862553 16278 log.cc:824] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:12.863865 16276 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: No bootstrap required, opened a new log
I20260620 09:08:12.864084 16276 ts_tablet_manager.cc:1397] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 09:08:12.864531 16277 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: No bootstrap required, opened a new log
I20260620 09:08:12.864723 16277 ts_tablet_manager.cc:1397] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 09:08:12.864843 16278 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: No bootstrap required, opened a new log
I20260620 09:08:12.865012 16278 ts_tablet_manager.cc:1397] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 09:08:12.867218 16276 raft_consensus.cc:348] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.867497 16276 raft_consensus.cc:374] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:12.867579 16276 raft_consensus.cc:729] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0874dbddb04923ad333adaade22785, State: Initialized, Role: FOLLOWER
I20260620 09:08:12.868057 16276 consensus_queue.cc:260] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.868692 16277 raft_consensus.cc:348] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.868692 16278 raft_consensus.cc:348] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.868947 16278 raft_consensus.cc:374] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:12.868947 16277 raft_consensus.cc:374] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:12.868996 16277 raft_consensus.cc:729] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e284ba9e1045b0b1f7552ffd9fb304, State: Initialized, Role: FOLLOWER
I20260620 09:08:12.868996 16278 raft_consensus.cc:729] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c10a16140024fe091049b2d2b672e3b, State: Initialized, Role: FOLLOWER
I20260620 09:08:12.869154 16276 ts_tablet_manager.cc:1428] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 09:08:12.869531 16259 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
I20260620 09:08:12.869481 16277 consensus_queue.cc:260] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.869481 16278 consensus_queue.cc:260] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.870393 16125 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
I20260620 09:08:12.870452 16278 ts_tablet_manager.cc:1428] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 09:08:12.870452 16277 ts_tablet_manager.cc:1428] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 09:08:12.870599 15991 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
I20260620 09:08:12.941933 16284 raft_consensus.cc:482] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:12.942147 16284 raft_consensus.cc:504] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.943243 16284 leader_election.cc:288] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831), 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529)
I20260620 09:08:12.947820 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.943985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.1:46043)
0620 09:08:12.944357 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.944373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.944479 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.944742 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.944744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.944761 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:12.944887 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.944894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.945717 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.945720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.946640 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.946653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.947338 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.947342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.947440 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.947480 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.947557 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.947645 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":20,"spinlock_wait_cycles":768,"thread_start_us":148,"threads_started":1}
I20260620 09:08:12.947801 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.943687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36757 (local address 127.0.98.3:33831)
0620 09:08:12.943819 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.943824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.943943 (+ 119us) server_negotiation.cc:407] Connection header received
0620 09:08:12.944220 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.944223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.944281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.944385 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:12.944739 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.945489 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.946671 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.947134 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.947215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.947295 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.947624 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.947699 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:08:12.947832 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.944097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46043 (local address 127.0.98.2:34529)
0620 09:08:12.944242 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.944247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.944446 (+ 199us) server_negotiation.cc:407] Connection header received
0620 09:08:12.944566 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.944569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.944623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.944728 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:12.944999 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.945611 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.946773 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.947212 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.947317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.947380 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.947618 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.947703 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:08:12.947857 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.943569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33831 (local address 127.0.98.1:36757)
0620 09:08:12.943839 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:12.943855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:12.943969 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:12.944412 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:12.944415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:12.944441 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:12.944579 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.944586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.945611 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.945616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.946551 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:12.946561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.947278 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.947285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:12.947382 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.947412 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:08:12.947599 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 09:08:12.947717 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":37,"spinlock_wait_cycles":3840,"thread_start_us":110,"threads_started":1}
I20260620 09:08:12.948635 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9423dd57df0d4703b4ddb77e45777f8e" candidate_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c10a16140024fe091049b2d2b672e3b" is_pre_election: true
I20260620 09:08:12.948611 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9423dd57df0d4703b4ddb77e45777f8e" candidate_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0874dbddb04923ad333adaade22785" is_pre_election: true
I20260620 09:08:12.948972 16213 raft_consensus.cc:2454] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e284ba9e1045b0b1f7552ffd9fb304 in term 0.
I20260620 09:08:12.948972 16079 raft_consensus.cc:2454] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e284ba9e1045b0b1f7552ffd9fb304 in term 0.
I20260620 09:08:12.949502 15879 leader_election.cc:302] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304, cf0874dbddb04923ad333adaade22785; no voters:
I20260620 09:08:12.949803 16284 raft_consensus.cc:2781] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:08:12.949931 16284 raft_consensus.cc:482] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:08:12.950023 16284 raft_consensus.cc:3037] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:12.951087 16284 raft_consensus.cc:504] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.951459 16284 leader_election.cc:288] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 1 election: Requested vote from peers cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831), 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529)
I20260620 09:08:12.951957 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9423dd57df0d4703b4ddb77e45777f8e" candidate_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c10a16140024fe091049b2d2b672e3b"
I20260620 09:08:12.951938 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9423dd57df0d4703b4ddb77e45777f8e" candidate_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0874dbddb04923ad333adaade22785"
I20260620 09:08:12.952081 16079 raft_consensus.cc:3037] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:12.952081 16213 raft_consensus.cc:3037] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:12.953476 16213 raft_consensus.cc:2454] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e284ba9e1045b0b1f7552ffd9fb304 in term 1.
I20260620 09:08:12.953476 16079 raft_consensus.cc:2454] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e284ba9e1045b0b1f7552ffd9fb304 in term 1.
I20260620 09:08:12.953821 15879 leader_election.cc:302] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304, 9c10a16140024fe091049b2d2b672e3b; no voters:
I20260620 09:08:12.954002 16284 raft_consensus.cc:2781] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:08:12.954232 16284 raft_consensus.cc:686] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 LEADER]: Becoming Leader. State: Replica: 92e284ba9e1045b0b1f7552ffd9fb304, State: Running, Role: LEADER
I20260620 09:08:12.954596 16284 consensus_queue.cc:237] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:12.957919 15799 catalog_manager.cc:5495] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } health_report { overall_health: UNKNOWN } } }
I20260620 09:08:12.969244 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:12.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.0.98.1:41639)
0620 09:08:12.965733 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:12.965739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:12.965753 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:08:12.965909 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:12.965911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:12.965968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:12.966078 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:08:12.966332 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.966838 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.967711 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:12.967934 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:12.968048 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:12.968371 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:08:12.968807 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:08:12.968956 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:12.968992 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 09:08:12.969087 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 09:08:12.975682 16213 raft_consensus.cc:1261] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Refusing update from remote peer 92e284ba9e1045b0b1f7552ffd9fb304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:08:12.975682 16079 raft_consensus.cc:1261] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Refusing update from remote peer 92e284ba9e1045b0b1f7552ffd9fb304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:08:12.976430 16284 consensus_queue.cc:1035] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:08:12.976728 16287 consensus_queue.cc:1035] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:08:12.985548 16292 mvcc.cc:205] Tried to move back new op lower bound from 7298852835219808256 to 7298852835144962048. Current Snapshot: MvccSnapshot[applied={T|T < 7298852835219808256}]
I20260620 09:08:12.985755 16291 mvcc.cc:205] Tried to move back new op lower bound from 7298852835219808256 to 7298852835144962048. Current Snapshot: MvccSnapshot[applied={T|T < 7298852835219808256}]
I20260620 09:08:12.994127 15799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:08:12.994356 15799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:08:12.997139 15799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Sending DeleteTablet for 3 replicas of tablet 9423dd57df0d4703b4ddb77e45777f8e
I20260620 09:08:12.998080 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42924
I20260620 09:08:12.998075 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40200
I20260620 09:08:12.998075 15925 tablet_service.cc:1514] Processing DeleteTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38420
I20260620 09:08:12.998667 16301 tablet_replica.cc:330] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: stopping tablet replica
I20260620 09:08:12.998696 16300 tablet_replica.cc:330] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: stopping tablet replica
I20260620 09:08:12.998874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15862
I20260620 09:08:12.999091 16300 raft_consensus.cc:2229] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:08:12.999620 16300 raft_consensus.cc:2258] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:08:13.000770 16301 raft_consensus.cc:2229] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:08:13.001199 16301 raft_consensus.cc:2258] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:08:13.001572 16300 ts_tablet_manager.cc:1905] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:08:13.002498 16301 ts_tablet_manager.cc:1905] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:08:13.004667 16301 ts_tablet_manager.cc:1918] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:08:13.004799 16301 log.cc:1196] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/9423dd57df0d4703b4ddb77e45777f8e
I20260620 09:08:13.005172 16301 ts_tablet_manager.cc:1939] T 9423dd57df0d4703b4ddb77e45777f8e P 9c10a16140024fe091049b2d2b672e3b: Deleting consensus metadata
I20260620 09:08:13.005203 16300 ts_tablet_manager.cc:1918] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:08:13.005319 16300 log.cc:1196] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/9423dd57df0d4703b4ddb77e45777f8e
I20260620 09:08:13.005672 16300 ts_tablet_manager.cc:1939] T 9423dd57df0d4703b4ddb77e45777f8e P cf0874dbddb04923ad333adaade22785: Deleting consensus metadata
I20260620 09:08:13.006541 15785 catalog_manager.cc:4841] TS 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): tablet 9423dd57df0d4703b4ddb77e45777f8e (table test-table [id=479e3aa099284b44adb96b648f1c42de]) successfully deleted
I20260620 09:08:13.006923 15785 catalog_manager.cc:4841] TS cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): tablet 9423dd57df0d4703b4ddb77e45777f8e (table test-table [id=479e3aa099284b44adb96b648f1c42de]) successfully deleted
W20260620 09:08:13.007381 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41639 (error 108)
I20260620 09:08:13.007601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15996
I20260620 09:08:13.008952 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:13.008664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41639 (local address 127.0.0.1:38436)
0620 09:08:13.008803 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:13.008875 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 09:08:13.009202 15786 catalog_manager.cc:4584] TS 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9423dd57df0d4703b4ddb77e45777f8e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41639: connect: Connection refused (error 111)
I20260620 09:08:13.014307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16130
I20260620 09:08:13.020181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 15768
I20260620 09:08:33.027125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--webserver_interface=127.0.98.62
--webserver_port=38583
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:33.141727 16304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:33.142030 16304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:33.142074 16304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:33.146282 16304 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:08:33.146347 16304 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:08:33.146369 16304 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:08:33.146384 16304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:08:33.146402 16304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:33.146420 16304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:33.146436 16304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:08:33.146481 16304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:08:33.152081 16304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:33.153257 16304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:33.154338 16304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:33.167899 16313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.167930 16311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.168172 16310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.168573 16304 server_base.cc:1034] running on GCE node
I20260620 09:08:33.169334 16304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:33.170691 16304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:33.172097 16304 hybrid_clock.cc:648] HybridClock initialized: now 1781946513172057 us; error 54 us; skew 500 ppm
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:33.179893 16304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:08:33.181010 16304 webserver.cc:460] Webserver started at http://127.0.98.62:38583/ using document root <none> and password file <none>
I20260620 09:08:33.181717 16304 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:33.181847 16304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:33.186300 16304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 09:08:33.188751 16320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.190168 16304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:08:33.190289 16304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "617b926feb6b463498573d88c1af5b9b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "8c66c4d462781bb3d34753a5ae972e4e"
server_key_iv: "fc6586362395fdfd581bcd93d3967c04"
server_key_version: "encryptionkey@0"
I20260620 09:08:33.190922 16304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:33.202257 16304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:33.206316 16304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:33.206506 16304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:33.214184 16304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33061
I20260620 09:08:33.214466 16372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33061 every 8 connection(s)
I20260620 09:08:33.215428 16304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:08:33.220311 16373 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:08:33.222152 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap starting.
I20260620 09:08:33.224166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16304
I20260620 09:08:33.224673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41639
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=40225
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:08:33.240715 16373 log.cc:824] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:33.250144 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:08:33.250608 16373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap complete.
I20260620 09:08:33.254884 16373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:33.255291 16373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Initialized, Role: FOLLOWER
I20260620 09:08:33.255834 16373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:33.255956 16373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:08:33.256042 16373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:08:33.256165 16373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 1 FOLLOWER]: Advancing to term 2
I20260620 09:08:33.258242 16373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:33.258596 16373 leader_election.cc:302] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617b926feb6b463498573d88c1af5b9b; no voters:
I20260620 09:08:33.258951 16373 leader_election.cc:288] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 09:08:33.259112 16377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 09:08:33.259328 16377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 LEADER]: Becoming Leader. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Running, Role: LEADER
I20260620 09:08:33.259725 16377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:33.260340 16373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:08:33.261344 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617b926feb6b463498573d88c1af5b9b. Latest consensus state: current_term: 2 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:33.261478 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
I20260620 09:08:33.261350 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:33.261759 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
I20260620 09:08:33.265908 16392 catalog_manager.cc:1259] Loaded cluster ID: 5b66b21584d24952886973dab2a30939
I20260620 09:08:33.265971 16392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: loading cluster ID for follower catalog manager: success
I20260620 09:08:33.268596 16392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: acquiring CA information for follower catalog manager: success
I20260620 09:08:33.270361 16392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:08:33.270569 16386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:08:33.273313 16386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:08:33.273821 16386 catalog_manager.cc:1259] Loaded cluster ID: 5b66b21584d24952886973dab2a30939
I20260620 09:08:33.273886 16386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:08:33.274561 16386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:08:33.275038 16386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Loaded TSK: 0
I20260620 09:08:33.288414 16386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Generated new TSK 1
I20260620 09:08:33.288695 16386 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:08:33.349027 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:33.349412 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:33.349463 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:33.353801 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:33.353878 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:33.353947 16375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:08:33.359601 16375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40225
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:33.360903 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:33.362263 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:33.372718 16400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.372902 16375 server_base.cc:1034] running on GCE node
W20260620 09:08:33.372718 16403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.373260 16401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.373692 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:33.374473 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:33.375667 16375 hybrid_clock.cc:648] HybridClock initialized: now 1781946513375641 us; error 44 us; skew 500 ppm
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:33.378662 16375 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:08:33.379873 16375 webserver.cc:460] Webserver started at http://127.0.98.1:40225/ using document root <none> and password file <none>
I20260620 09:08:33.380683 16375 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:33.380787 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:33.385215 16375 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:08:33.387832 16410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.389401 16375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:08:33.389570 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "4f7c282c646e49582ada06c8cb060803"
server_key_iv: "55c6e8e36865b5d4716a2bdda55697bc"
server_key_version: "encryptionkey@0"
I20260620 09:08:33.390180 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:33.416123 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:33.420882 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:33.421144 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:33.421813 16375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:33.422991 16417 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:08:33.427511 16375 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:08:33.427623 16375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260620 09:08:33.427702 16375 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:08:33.429840 16375 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:08:33.429906 16375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 09:08:33.430167 16417 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap starting.
I20260620 09:08:33.441264 16375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41639
I20260620 09:08:33.441290 16524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41639 every 8 connection(s)
I20260620 09:08:33.442615 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:08:33.451506 16417 log.cc:824] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:33.452330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16375
I20260620 09:08:33.452818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34529
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:08:33.460551 16417 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:08:33.461115 16417 tablet_bootstrap.cc:492] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap complete.
I20260620 09:08:33.462569 16417 ts_tablet_manager.cc:1397] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.009s
I20260620 09:08:33.462729 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.445151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:49455)
0620 09:08:33.445722 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.445765 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.446628 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.447760 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.447768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.448196 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:33.448671 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.448685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.450085 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.450088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.450949 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.450958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.451731 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.451735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.451949 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.452688 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:33.452712 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:33.454821 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:33.458578 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.458584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.458597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.458972 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.459466 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.459469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.459471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.459643 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.460492 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:33.460499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:33.460824 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.461166 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.461594 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":167,"threads_started":1}
I20260620 09:08:33.463299 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.445141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49455 (local address 127.0.98.62:33061)
0620 09:08:33.445535 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.445557 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.445889 (+ 332us) server_negotiation.cc:407] Connection header received
0620 09:08:33.447061 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.447072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.447365 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.447701 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.448848 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.449900 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.451077 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.451555 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.451741 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.455101 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.455112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.455158 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.455205 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.457695 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.459117 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.459127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.459136 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.459341 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.459755 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.459760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.459763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.460354 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.460517 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.461382 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.461732 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":128,"threads_started":1}
I20260620 09:08:33.465186 16417 raft_consensus.cc:348] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:33.465783 16417 raft_consensus.cc:729] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e284ba9e1045b0b1f7552ffd9fb304, State: Initialized, Role: FOLLOWER
I20260620 09:08:33.466121 16525 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:33.466367 16525 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:33.466339 16417 consensus_queue.cc:260] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } } peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } }
I20260620 09:08:33.466918 16525 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:33.467352 16417 ts_tablet_manager.cc:1428] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:08:33.469489 16337 ts_manager.cc:194] Registered new tserver with Master: 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639)
I20260620 09:08:33.473143 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49455
I20260620 09:08:33.475898 16525 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 09:08:33.483893 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.472338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41639 (local address 127.0.0.1:38654)
0620 09:08:33.472929 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.472977 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.473158 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.473702 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.473708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.474189 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:33.474320 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.474327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.475651 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.475655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.476692 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.476705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.477623 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.477646 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.477757 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.478325 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:33.478360 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:33.480037 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:33.481960 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.481972 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.481978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.482229 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.482551 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.482555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.482557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.482672 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.483177 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:33.483183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:33.483323 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.483465 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.483701 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":125,"threads_started":1}
I20260620 09:08:33.483893 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.472434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.0.98.1:41639)
0620 09:08:33.473165 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.473180 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.473206 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:08:33.473327 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.473333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.473536 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.473691 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.474457 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.475520 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.476823 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.477384 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.477771 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.480195 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.480214 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.480225 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.480247 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.481840 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.482363 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.482368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.482369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.482452 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.482769 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.482774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.482776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.483034 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.483209 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.483560 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.483731 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":630,"thread_start_us":93,"threads_started":1}
I20260620 09:08:33.484968 16459 tablet_service.cc:1514] Processing DeleteTablet for tablet 9423dd57df0d4703b4ddb77e45777f8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38654
I20260620 09:08:33.486081 16534 tablet_replica.cc:330] stopping tablet replica
I20260620 09:08:33.486394 16534 raft_consensus.cc:2229] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:08:33.486665 16534 raft_consensus.cc:2258] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:08:33.487660 16534 ts_tablet_manager.cc:1905] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:08:33.491449 16534 ts_tablet_manager.cc:1918] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:08:33.491564 16534 log.cc:1196] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal/wals/9423dd57df0d4703b4ddb77e45777f8e
I20260620 09:08:33.491863 16534 ts_tablet_manager.cc:1939] T 9423dd57df0d4703b4ddb77e45777f8e P 92e284ba9e1045b0b1f7552ffd9fb304: Deleting consensus metadata
I20260620 09:08:33.493687 16323 catalog_manager.cc:4841] TS 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639): tablet 9423dd57df0d4703b4ddb77e45777f8e (table test-table [id=479e3aa099284b44adb96b648f1c42de]) successfully deleted
W20260620 09:08:33.575932 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:33.576237 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:33.576282 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:33.580714 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:33.580793 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:33.580857 16530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:08:33.586508 16530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:33.587761 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:33.588897 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:33.599275 16539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.599293 16540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.599293 16542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.599606 16530 server_base.cc:1034] running on GCE node
I20260620 09:08:33.600126 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:33.600822 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:33.602025 16530 hybrid_clock.cc:648] HybridClock initialized: now 1781946513602012 us; error 61 us; skew 500 ppm
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:33.604849 16530 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:08:33.605801 16530 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260620 09:08:33.606352 16530 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:33.606426 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:33.610246 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:08:33.612315 16549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.613716 16530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:08:33.613858 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "9c10a16140024fe091049b2d2b672e3b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "68b0ababb362bfb51f3827f9e397a407"
server_key_iv: "b42184a17b8c73477cd61dc9ac4db644"
server_key_version: "encryptionkey@0"
I20260620 09:08:33.614425 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:33.630802 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:33.634196 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:33.634517 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:33.635170 16530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:33.636029 16530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:33.636103 16530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.636179 16530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:33.636224 16530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.644951 16530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260620 09:08:33.645015 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260620 09:08:33.646070 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:08:33.651544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16530
I20260620 09:08:33.652086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33831
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=36313
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:08:33.661823 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.648028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59509 (local address 127.0.98.62:33061)
0620 09:08:33.648174 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.648178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.648994 (+ 816us) server_negotiation.cc:407] Connection header received
0620 09:08:33.650360 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.650364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.650430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.650537 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.651864 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.652518 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.654139 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.654381 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.654507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.657627 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.657631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.657633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.657662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.659183 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.659902 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.659905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.659906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.659995 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.660439 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.660441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.660443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.660635 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.660736 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.661554 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.661693 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:08:33.663213 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.648358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:59509)
0620 09:08:33.648830 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.648872 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.650141 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.650740 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.650747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.651116 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:33.651632 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.651646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.652650 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.652654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.653994 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.654009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.654568 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.654575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.654787 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.655414 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:33.655431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:33.657406 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:33.659319 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.659328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.659351 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.659782 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.660106 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.660110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.660114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.660331 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.660768 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:33.660776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:33.661227 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.661482 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.661936 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260620 09:08:33.664579 16663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:33.664847 16663 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:33.665447 16663 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:33.666679 16337 ts_manager.cc:194] Registered new tserver with Master: 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529)
I20260620 09:08:33.667392 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59509
W20260620 09:08:33.771050 16666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:33.771363 16666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:33.771452 16666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:33.775650 16666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:33.775750 16666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:33.775841 16666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:08:33.781634 16666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36313
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:33.782840 16666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:33.783923 16666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:33.794116 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:33.794051 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.794240 16666 server_base.cc:1034] running on GCE node
W20260620 09:08:33.794051 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:33.795014 16666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:33.795814 16666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:33.796988 16666 hybrid_clock.cc:648] HybridClock initialized: now 1781946513796938 us; error 54 us; skew 500 ppm
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:33.799885 16666 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:08:33.800786 16666 webserver.cc:460] Webserver started at http://127.0.98.3:36313/ using document root <none> and password file <none>
I20260620 09:08:33.801424 16666 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:33.801539 16666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:33.805439 16666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:08:33.807603 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.808936 16666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:08:33.809118 16666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "cf0874dbddb04923ad333adaade22785"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "63dc78f340de0a6475357ac02cbfbe61"
server_key_iv: "08232457c50cdd3ed24bb38af190dfa1"
server_key_version: "encryptionkey@0"
I20260620 09:08:33.809701 16666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:33.831453 16666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:33.834934 16666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:33.835165 16666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:33.835850 16666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:33.836719 16666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:33.836793 16666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.836866 16666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:33.836908 16666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:33.846458 16666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33831
I20260620 09:08:33.846484 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33831 every 8 connection(s)
I20260620 09:08:33.847500 16666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:08:33.849130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16666
Jun 20 09:08:33 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:08:33.858860 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.849772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.98.62:33061)
0620 09:08:33.850011 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.850015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.850190 (+ 175us) server_negotiation.cc:407] Connection header received
0620 09:08:33.850346 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.850349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.850403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.850497 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.851488 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.852082 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.853354 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.853625 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.853722 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.855270 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.855273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.855276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.855303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.857110 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.857572 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.857575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.857577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.857662 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.857949 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.857954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.857956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.858186 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.858410 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.858546 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.858675 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260620 09:08:33.863010 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.849412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45911 (local address 127.0.98.62:33061)
0620 09:08:33.849574 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.849579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.850690 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 09:08:33.852262 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.852265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.852310 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.852388 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.853853 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.854338 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.855750 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.855977 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.856092 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.858722 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.858726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.858727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.858774 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.860520 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.861318 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.861322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.861324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.861415 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.861839 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.861843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.861844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.862054 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.862169 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.862777 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.862880 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:08:33.863991 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.849963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:45911)
0620 09:08:33.850532 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.850588 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.852047 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.852577 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.852586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.853098 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:33.853732 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.853751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.854491 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.854495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.855614 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.855625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.856141 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.856146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.856339 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.856908 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:33.856925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:33.858536 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:33.860661 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.860668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.860686 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.861196 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.861531 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:33.861536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:33.861538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:33.861722 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:33.862181 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:33.862190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:33.862494 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.862712 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.863031 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":162,"threads_started":1}
I20260620 09:08:33.865265 16795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33061
I20260620 09:08:33.865586 16795 heartbeater.cc:464] Registering TS with master...
I20260620 09:08:33.866137 16795 heartbeater.cc:511] Master 127.0.98.62:33061 requested a full tablet report, sending...
I20260620 09:08:33.867192 16337 ts_manager.cc:194] Registered new tserver with Master: cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
I20260620 09:08:33.867784 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45911
I20260620 09:08:33.872826 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:08:33.877357 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.873589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.62:33061)
0620 09:08:33.873755 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.873761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.873916 (+ 155us) server_negotiation.cc:407] Connection header received
0620 09:08:33.874134 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.874137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.874244 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.874373 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:08:33.874674 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.875198 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.876109 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.876328 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.876445 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.876768 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:08:33.876910 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 09:08:33.877213 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54608: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 09:08:33.877450 16527 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54608: token expired
W20260620 09:08:33.877727 16262 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33061)
I20260620 09:08:33.884914 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.878089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.98.62:33061)
0620 09:08:33.878197 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.878200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.878213 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:08:33.878325 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.878327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.878364 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.878426 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:33.879019 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.879548 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.880375 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.880607 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.880696 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.882156 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:33.882159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:33.882161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:33.882182 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:33.883512 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.883893 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.883896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.883897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.883972 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:33.884227 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:33.884229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:33.884230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:33.884393 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:33.884530 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.884702 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.884792 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:08:33.887501 16337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:08:33.890933 16337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:08:33.899427 16459 tablet_service.cc:1467] Processing CreateTablet for tablet 020a34f51dbd4495a108fc542f412fb9 (DEFAULT_TABLE table=test-table [id=50c115be8e574b8bb57c075bd392120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:33.900046 16459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020a34f51dbd4495a108fc542f412fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:33.903067 16417 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap starting.
I20260620 09:08:33.903770 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.898384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33831 (local address 127.0.0.1:48490)
0620 09:08:33.898709 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.898723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.898819 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.899466 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.899470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.899488 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:33.899620 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.899627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.901406 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.901409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.902380 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.902390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.903302 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.903306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.903398 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.903449 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.903536 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.903616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260620 09:08:33.903820 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.898471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.0.98.3:33831)
0620 09:08:33.898980 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.898994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.899018 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:08:33.899107 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.899118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.899310 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.899489 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:33.899759 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.901161 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.902518 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.903193 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.903300 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.903451 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.903583 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.903679 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":175,"threads_started":1}
I20260620 09:08:33.904042 16417 tablet_bootstrap.cc:654] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:33.904840 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.898489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.2:34529)
0620 09:08:33.899835 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.899860 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.899884 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:08:33.899955 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.899960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.900158 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.900313 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:33.900530 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.901657 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.903093 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.904205 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.904338 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.904493 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.904657 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.904731 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":82,"threads_started":1}
I20260620 09:08:33.904835 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.898058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.0.1:48982)
0620 09:08:33.898175 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.898192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.898281 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.900311 (+ 2030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.900314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.900334 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:33.900428 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.900435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.901779 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.901785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.902934 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.902945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.904347 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.904353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.904506 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.904541 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.904626 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.904722 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":23}
I20260620 09:08:33.906268 16417 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: No bootstrap required, opened a new log
I20260620 09:08:33.906355 16417 ts_tablet_manager.cc:1397] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 09:08:33.906801 16417 raft_consensus.cc:348] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.906924 16417 raft_consensus.cc:374] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:33.906960 16417 raft_consensus.cc:729] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e284ba9e1045b0b1f7552ffd9fb304, State: Initialized, Role: FOLLOWER
I20260620 09:08:33.907080 16417 consensus_queue.cc:260] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.907330 16417 ts_tablet_manager.cc:1428] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 09:08:33.907981 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 020a34f51dbd4495a108fc542f412fb9 (DEFAULT_TABLE table=test-table [id=50c115be8e574b8bb57c075bd392120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:33.908015 16597 tablet_service.cc:1467] Processing CreateTablet for tablet 020a34f51dbd4495a108fc542f412fb9 (DEFAULT_TABLE table=test-table [id=50c115be8e574b8bb57c075bd392120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:08:33.909094 16597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020a34f51dbd4495a108fc542f412fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:33.909096 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020a34f51dbd4495a108fc542f412fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:08:33.916214 16809 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Bootstrap starting.
I20260620 09:08:33.917377 16808 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Bootstrap starting.
I20260620 09:08:33.918660 16809 tablet_bootstrap.cc:654] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:33.919745 16808 tablet_bootstrap.cc:654] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Neither blocks nor log segments found. Creating new log.
I20260620 09:08:33.920032 16809 log.cc:824] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:33.920483 16808 log.cc:824] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:33.922098 16809 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: No bootstrap required, opened a new log
I20260620 09:08:33.922178 16808 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: No bootstrap required, opened a new log
I20260620 09:08:33.922338 16809 ts_tablet_manager.cc:1397] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 09:08:33.922363 16808 ts_tablet_manager.cc:1397] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:08:33.925573 16808 raft_consensus.cc:348] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.925825 16808 raft_consensus.cc:374] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:33.925892 16808 raft_consensus.cc:729] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c10a16140024fe091049b2d2b672e3b, State: Initialized, Role: FOLLOWER
I20260620 09:08:33.925761 16809 raft_consensus.cc:348] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.926095 16809 raft_consensus.cc:374] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:08:33.926175 16809 raft_consensus.cc:729] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0874dbddb04923ad333adaade22785, State: Initialized, Role: FOLLOWER
I20260620 09:08:33.926416 16808 consensus_queue.cc:260] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.926579 16809 consensus_queue.cc:260] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.927217 16663 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
I20260620 09:08:33.927256 16795 heartbeater.cc:503] Master 127.0.98.62:33061 was elected leader, sending a full tablet report...
I20260620 09:08:33.927542 16808 ts_tablet_manager.cc:1428] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:08:33.927615 16809 ts_tablet_manager.cc:1428] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 09:08:33.943949 16526 tablet.cc:2369] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:08:33.955962 16813 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:33.956183 16813 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.957437 16813 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639)
I20260620 09:08:33.961476 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.957777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48169 (local address 127.0.98.2:34529)
0620 09:08:33.957912 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.957917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.957936 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:08:33.958091 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.958093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.958136 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.958225 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:33.958485 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.959028 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.960299 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.960927 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.961012 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.961165 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.961309 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.961371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:08:33.961549 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.957651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.3:48169)
0620 09:08:33.957856 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.957875 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.957973 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.958234 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.958237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.958260 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:33.958376 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.958381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.959151 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.959157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.960190 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.960199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.961071 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.961075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.961183 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.961210 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.961288 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.961371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 09:08:33.962211 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.957972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41113 (local address 127.0.98.1:41639)
0620 09:08:33.958096 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.958101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.958206 (+ 105us) server_negotiation.cc:407] Connection header received
0620 09:08:33.958359 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.958362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.958423 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.958544 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:08:33.958800 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.959520 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.960823 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.961475 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.961581 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.961790 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.962052 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.962117 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:08:33.962237 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.957722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41639 (local address 127.0.98.3:41113)
0620 09:08:33.958101 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:33.958116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:33.958257 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:33.958554 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:33.958557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:33.958573 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:08:33.958694 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.958700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.959645 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.959648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.960703 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:33.960713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.961738 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.961742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:33.961894 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.961922 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 09:08:33.962027 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 09:08:33.962096 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":145,"threads_started":1}
I20260620 09:08:33.962353 16617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020a34f51dbd4495a108fc542f412fb9" candidate_uuid: "cf0874dbddb04923ad333adaade22785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c10a16140024fe091049b2d2b672e3b" is_pre_election: true
I20260620 09:08:33.962725 16617 raft_consensus.cc:2454] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0874dbddb04923ad333adaade22785 in term 0.
I20260620 09:08:33.962823 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020a34f51dbd4495a108fc542f412fb9" candidate_uuid: "cf0874dbddb04923ad333adaade22785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" is_pre_election: true
I20260620 09:08:33.963114 16479 raft_consensus.cc:2454] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0874dbddb04923ad333adaade22785 in term 0.
I20260620 09:08:33.963253 16683 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785; no voters:
I20260620 09:08:33.963546 16813 raft_consensus.cc:2781] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:08:33.963649 16813 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:08:33.963716 16813 raft_consensus.cc:3037] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:33.964982 16813 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.965379 16813 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [CANDIDATE]: Term 1 election: Requested vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639)
I20260620 09:08:33.965754 16617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020a34f51dbd4495a108fc542f412fb9" candidate_uuid: "cf0874dbddb04923ad333adaade22785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c10a16140024fe091049b2d2b672e3b"
I20260620 09:08:33.965763 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020a34f51dbd4495a108fc542f412fb9" candidate_uuid: "cf0874dbddb04923ad333adaade22785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
I20260620 09:08:33.965885 16479 raft_consensus.cc:3037] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:33.965885 16617 raft_consensus.cc:3037] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:08:33.967187 16617 raft_consensus.cc:2454] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0874dbddb04923ad333adaade22785 in term 1.
I20260620 09:08:33.967187 16479 raft_consensus.cc:2454] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0874dbddb04923ad333adaade22785 in term 1.
I20260620 09:08:33.967499 16684 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304, cf0874dbddb04923ad333adaade22785; no voters:
I20260620 09:08:33.967698 16813 raft_consensus.cc:2781] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:08:33.967938 16813 raft_consensus.cc:686] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 1 LEADER]: Becoming Leader. State: Replica: cf0874dbddb04923ad333adaade22785, State: Running, Role: LEADER
I20260620 09:08:33.968302 16813 consensus_queue.cc:237] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:33.971537 16337 catalog_manager.cc:5495] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf0874dbddb04923ad333adaade22785 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cf0874dbddb04923ad333adaade22785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260620 09:08:33.988286 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:33.984632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.98.3:33831)
0620 09:08:33.984771 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:33.984776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:33.984791 (+ 15us) server_negotiation.cc:407] Connection header received
0620 09:08:33.984909 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:33.984911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:33.984989 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:33.985159 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:08:33.985433 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.985951 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.986798 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:33.987025 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:33.987134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:33.987444 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:08:33.987872 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:08:33.988012 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:33.988069 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 09:08:33.988131 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:08:33.994889 16479 raft_consensus.cc:1261] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Refusing update from remote peer cf0874dbddb04923ad333adaade22785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:08:33.994910 16617 raft_consensus.cc:1261] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Refusing update from remote peer cf0874dbddb04923ad333adaade22785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:08:33.995661 16815 consensus_queue.cc:1035] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:08:33.995878 16813 consensus_queue.cc:1035] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:08:34.001729 16818 mvcc.cc:205] Tried to move back new op lower bound from 7298852921313853440 to 7298852921216741376. Current Snapshot: MvccSnapshot[applied={T|T < 7298852921313853440}]
I20260620 09:08:34.011482 16337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:08:34.011680 16337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:08:34.014226 16337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Sending DeleteTablet for 3 replicas of tablet 020a34f51dbd4495a108fc542f412fb9
I20260620 09:08:34.015100 16459 tablet_service.cc:1514] Processing DeleteTablet for tablet 020a34f51dbd4495a108fc542f412fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38654
I20260620 09:08:34.015151 16597 tablet_service.cc:1514] Processing DeleteTablet for tablet 020a34f51dbd4495a108fc542f412fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48982
I20260620 09:08:34.015764 16829 tablet_replica.cc:330] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: stopping tablet replica
I20260620 09:08:34.016003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16375
I20260620 09:08:34.015954 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 020a34f51dbd4495a108fc542f412fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:08:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48490
I20260620 09:08:34.016197 16829 raft_consensus.cc:2229] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:08:34.016844 16829 raft_consensus.cc:2258] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:08:34.018358 16830 tablet_replica.cc:330] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: stopping tablet replica
I20260620 09:08:34.018810 16830 raft_consensus.cc:2229] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:08:34.019225 16829 ts_tablet_manager.cc:1905] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:08:34.019358 16830 raft_consensus.cc:2258] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:08:34.022073 16830 ts_tablet_manager.cc:1905] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:08:34.023125 16829 ts_tablet_manager.cc:1918] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:08:34.023346 16829 log.cc:1196] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/020a34f51dbd4495a108fc542f412fb9
I20260620 09:08:34.023734 16829 ts_tablet_manager.cc:1939] T 020a34f51dbd4495a108fc542f412fb9 P 9c10a16140024fe091049b2d2b672e3b: Deleting consensus metadata
I20260620 09:08:34.025300 16830 ts_tablet_manager.cc:1918] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:08:34.025293 16322 catalog_manager.cc:4841] TS 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): tablet 020a34f51dbd4495a108fc542f412fb9 (table test-table [id=50c115be8e574b8bb57c075bd392120f]) successfully deleted
I20260620 09:08:34.025449 16830 log.cc:1196] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/020a34f51dbd4495a108fc542f412fb9
I20260620 09:08:34.025765 16830 ts_tablet_manager.cc:1939] T 020a34f51dbd4495a108fc542f412fb9 P cf0874dbddb04923ad333adaade22785: Deleting consensus metadata
W20260620 09:08:34.026314 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41639 (error 108)
I20260620 09:08:34.027020 16322 catalog_manager.cc:4841] TS cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): tablet 020a34f51dbd4495a108fc542f412fb9 (table test-table [id=50c115be8e574b8bb57c075bd392120f]) successfully deleted
W20260620 09:08:34.027570 16532 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:08:34.027298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41639 (local address 127.0.0.1:38658)
0620 09:08:34.027433 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:34.027497 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41639: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 09:08:34.027731 16323 catalog_manager.cc:4584] TS 92e284ba9e1045b0b1f7552ffd9fb304 (127.0.98.1:41639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 020a34f51dbd4495a108fc542f412fb9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41639: connect: Connection reset by peer (error 104)
I20260620 09:08:34.028597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16530
I20260620 09:08:34.034941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16666
I20260620 09:08:34.040496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16304
I20260620 09:08:54.047315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--webserver_interface=127.0.98.62
--webserver_port=38583
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:08:54.167222 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:54.167526 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:54.167569 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:54.171919 16832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 09:08:54.171993 16832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 09:08:54.172009 16832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 09:08:54.172024 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:08:54.172043 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:54.172112 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:54.172144 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:08:54.172178 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:08:54.177948 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:54.179152 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:54.180281 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:54.190110 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.190133 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.190133 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:54.190208 16832 server_base.cc:1034] running on GCE node
I20260620 09:08:54.191069 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:54.192209 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:54.193379 16832 hybrid_clock.cc:648] HybridClock initialized: now 1781946534193345 us; error 47 us; skew 500 ppm
Jun 20 09:08:54 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:54.196167 16832 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:08:54.197218 16832 webserver.cc:460] Webserver started at http://127.0.98.62:38583/ using document root <none> and password file <none>
I20260620 09:08:54.197801 16832 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:54.197870 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:54.202015 16832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:08:54.204300 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.205672 16832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:08:54.205806 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "617b926feb6b463498573d88c1af5b9b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "8c66c4d462781bb3d34753a5ae972e4e"
server_key_iv: "fc6586362395fdfd581bcd93d3967c04"
server_key_version: "encryptionkey@0"
I20260620 09:08:54.206390 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:54.244930 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:54.248132 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:54.248355 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:54.256700 16832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33061
I20260620 09:08:54.257100 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33061 every 8 connection(s)
I20260620 09:08:54.258036 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:08:54.262599 16900 sys_catalog.cc:263] Verifying existing consensus state
I20260620 09:08:54.264389 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap starting.
I20260620 09:08:54.265288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16832
I20260620 09:08:54.265808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41639
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=40225
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:08:54.283566 16900 log.cc:824] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:54.295368 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:08:54.295805 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Bootstrap complete.
I20260620 09:08:54.300007 16900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:54.300519 16900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Initialized, Role: FOLLOWER
I20260620 09:08:54.301115 16900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:54.301240 16900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:08:54.301329 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:08:54.301458 16900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 2 FOLLOWER]: Advancing to term 3
I20260620 09:08:54.303644 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:54.304040 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617b926feb6b463498573d88c1af5b9b; no voters:
I20260620 09:08:54.304463 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 09:08:54.304560 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 09:08:54.304776 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [term 3 LEADER]: Becoming Leader. State: Replica: 617b926feb6b463498573d88c1af5b9b, State: Running, Role: LEADER
I20260620 09:08:54.305197 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } }
I20260620 09:08:54.305864 16900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:08:54.306712 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:54.306841 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
I20260620 09:08:54.307453 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617b926feb6b463498573d88c1af5b9b. Latest consensus state: current_term: 3 leader_uuid: "617b926feb6b463498573d88c1af5b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617b926feb6b463498573d88c1af5b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } }
I20260620 09:08:54.307564 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b [sys.catalog]: This master's current role is: LEADER
I20260620 09:08:54.307857 16913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:08:54.312685 16913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:08:54.313316 16913 catalog_manager.cc:1259] Loaded cluster ID: 5b66b21584d24952886973dab2a30939
I20260620 09:08:54.313390 16913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:08:54.315181 16913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:08:54.315811 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Loaded TSK: 0
I20260620 09:08:54.315901 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 617b926feb6b463498573d88c1af5b9b: Loaded TSK: 1
F20260620 09:08:54.316633 16913 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6fcffc59ad google::LogMessage::Fail() at ??:0
@ 0x7f6fcffc888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6fcffc54de google::LogMessage::Flush() at ??:0
@ 0x7f6fcffc6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6fd2c8918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6fd2c81f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6fda92a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6fda90499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6fda905178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6fda946df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6fda96582b std::function<>::operator()() at ??:0
@ 0x7f6fda904b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6fda905f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6fda902579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6fda946640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6fd22173d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6fd2217c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6fd22193f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6fd2208512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6fd384c6db start_thread at ??:0
@ 0x7f6fceac361f clone at ??:0
W20260620 09:08:54.422379 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:54.422667 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:54.422714 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:54.426971 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:54.427045 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:54.427109 16902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:08:54.432758 16902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40225
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:54.433959 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:54.435063 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:54.446779 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.446839 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.446779 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:54.447067 16902 server_base.cc:1034] running on GCE node
I20260620 09:08:54.447484 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:54.448169 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:54.449363 16902 hybrid_clock.cc:648] HybridClock initialized: now 1781946534449335 us; error 46 us; skew 500 ppm
Jun 20 09:08:54 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:54.452292 16902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:08:54.453353 16902 webserver.cc:460] Webserver started at http://127.0.98.1:40225/ using document root <none> and password file <none>
I20260620 09:08:54.453949 16902 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:54.454010 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:54.458210 16902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:08:54.460698 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.462082 16902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:08:54.462224 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "92e284ba9e1045b0b1f7552ffd9fb304"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "4f7c282c646e49582ada06c8cb060803"
server_key_iv: "55c6e8e36865b5d4716a2bdda55697bc"
server_key_version: "encryptionkey@0"
I20260620 09:08:54.462860 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:54.492203 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:54.495744 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:54.495978 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:54.496627 16902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:54.497907 16964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 09:08:54.503677 16902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 09:08:54.503882 16902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 09:08:54.503999 16902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 09:08:54.505973 16902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 09:08:54.506054 16902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 09:08:54.506382 16964 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap starting.
I20260620 09:08:54.517117 16902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41639
I20260620 09:08:54.517135 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41639 every 8 connection(s)
I20260620 09:08:54.518141 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:08:54.523940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 16902
I20260620 09:08:54.524386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34529
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:08:54.528910 16964 log.cc:824] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Log is configured to *not* fsync() on all Append() calls
I20260620 09:08:54.538501 16964 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 09:08:54.538954 16964 tablet_bootstrap.cc:492] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Bootstrap complete.
I20260620 09:08:54.540421 16964 ts_tablet_manager.cc:1397] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20260620 09:08:54.544612 16964 raft_consensus.cc:348] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:54.545425 16964 raft_consensus.cc:729] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e284ba9e1045b0b1f7552ffd9fb304, State: Initialized, Role: FOLLOWER
I20260620 09:08:54.546013 16964 consensus_queue.cc:260] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:54.547217 16964 ts_tablet_manager.cc:1428] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260620 09:08:54.550275 17074 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:08:54.520181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:47329)
0620 09:08:54.520686 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.520742 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:54.521597 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:54.549362 (+ 27765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":158,"threads_started":1}
W20260620 09:08:54.550755 17072 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33061 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33061: Client connection negotiation failed: client connection to 127.0.98.62:33061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 09:08:54.551685 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.551214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:50775)
0620 09:08:54.551415 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.551559 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:08:54.552414 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.552207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:46021)
0620 09:08:54.552315 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.552365 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 09:08:54.552664 17072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:08:54.641947 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:54.642197 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:54.642241 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:54.646437 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:54.646505 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:54.646595 17075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:08:54.652089 17075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:54.653311 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:54.654422 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:54.664383 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:54.664582 17075 server_base.cc:1034] running on GCE node
W20260620 09:08:54.664383 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.664383 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:54.665295 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:54.666172 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:54.667366 17075 hybrid_clock.cc:648] HybridClock initialized: now 1781946534667347 us; error 32 us; skew 500 ppm
Jun 20 09:08:54 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:54.670225 17075 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:08:54.671224 17075 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260620 09:08:54.671823 17075 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:54.671916 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:54.675923 17075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:08:54.678197 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.679462 17075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:08:54.679603 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "9c10a16140024fe091049b2d2b672e3b"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "68b0ababb362bfb51f3827f9e397a407"
server_key_iv: "b42184a17b8c73477cd61dc9ac4db644"
server_key_version: "encryptionkey@0"
I20260620 09:08:54.680186 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:54.691624 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:54.694793 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:54.695010 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:54.695653 17075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:54.696533 17075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:54.696605 17075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.696678 17075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:54.696722 17075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.706411 17075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34529
I20260620 09:08:54.706434 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34529 every 8 connection(s)
I20260620 09:08:54.707501 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:08:54.711169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17075
I20260620 09:08:54.711601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33831
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33061
--webserver_port=36313
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:08:54.711736 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.709796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:51333)
0620 09:08:54.710281 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.710401 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
W20260620 09:08:54.712248 17206 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33061 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33061: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
I20260620 09:08:54.712929 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.712705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:35325)
0620 09:08:54.712824 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.712874 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:08:54.713609 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.713410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:46083)
0620 09:08:54.713509 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.713563 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:08:54.713819 17206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 09:08:54.842780 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:08:54.843132 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:08:54.843242 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:08:54.847484 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:08:54.847594 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:08:54.847698 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:08:54.853333 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36313
--tserver_master_addrs=127.0.98.62:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:08:54.854605 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:08:54.855697 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:08:54.865813 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.865834 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:08:54.865836 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:08:54.866226 17209 server_base.cc:1034] running on GCE node
I20260620 09:08:54.866717 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:08:54.867426 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:08:54.868588 17209 hybrid_clock.cc:648] HybridClock initialized: now 1781946534868552 us; error 54 us; skew 500 ppm
Jun 20 09:08:54 dist-test-slave-92t4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:08:54.871439 17209 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:08:54.872399 17209 webserver.cc:460] Webserver started at http://127.0.98.3:36313/ using document root <none> and password file <none>
I20260620 09:08:54.872933 17209 fs_manager.cc:359] Metadata directory not provided
I20260620 09:08:54.872993 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:08:54.876904 17209 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:08:54.879070 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.880330 17209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:08:54.880453 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "cf0874dbddb04923ad333adaade22785"
format_stamp: "Formatted at 2026-06-20 09:08:12 on dist-test-slave-92t4"
server_key: "63dc78f340de0a6475357ac02cbfbe61"
server_key_iv: "08232457c50cdd3ed24bb38af190dfa1"
server_key_version: "encryptionkey@0"
I20260620 09:08:54.881000 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:08:54.907032 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:08:54.910333 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:08:54.910552 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:08:54.911203 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:08:54.912072 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:08:54.912119 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.912189 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:08:54.912230 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:08:54.922050 17209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33831
I20260620 09:08:54.922081 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33831 every 8 connection(s)
I20260620 09:08:54.923120 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:08:54.927153 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.925423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:58247)
0620 09:08:54.925886 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.925997 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
W20260620 09:08:54.927712 17338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33061 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33061: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
I20260620 09:08:54.928320 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.928116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:48525)
0620 09:08:54.928221 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.928271 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 09:08:54.928890 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:54.928710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:53215)
0620 09:08:54.928805 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:54.928843 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:08:54.929100 17338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 09:08:54.929344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17209
I20260620 09:08:55.553959 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:55.553502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:39779)
0620 09:08:55.553777 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:55.553845 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260620 09:08:55.714797 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:55.714396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:39317)
0620 09:08:55.714632 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:55.714690 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260620 09:08:55.930001 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:55.929605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:59157)
0620 09:08:55.929845 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:55.929897 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260620 09:08:56.345099 17345 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:56.345347 17345 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:56.346647 17345 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
Jun 20 09:08:56 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 09:08:56 dist-test-slave-92t4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 09:08:56.365854 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.347071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38493 (local address 127.0.98.2:34529)
0620 09:08:56.347541 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:56.347558 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:56.347604 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:08:56.348358 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:56.348364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:56.348745 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:56.349082 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:56.350371 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.351328 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.352500 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.352965 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.353167 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:56.361193 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:56.361198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:56.361212 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:56.361244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:56.363154 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363713 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:56.363723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:56.363727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:56.363846 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:56.364188 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:56.364197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:56.364199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:56.364616 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:56.364766 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:56.365361 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 09:08:56.365631 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":83,"threads_started":1}
I20260620 09:08:56.365851 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.346903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34529 (local address 127.0.98.1:38493)
0620 09:08:56.347291 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:56.347317 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:56.347440 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:56.349135 (+ 1695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:56.349141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:56.349698 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:56.350192 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:56.350207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.351491 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.351496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:56.352370 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:56.352379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.353116 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.353119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:56.353322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:56.358999 (+ 5677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:56.359013 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:56.360953 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:56.363283 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:56.363291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:56.363593 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:56.363946 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:56.363952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:56.364083 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:56.364741 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:56.364747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:56.364984 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 09:08:56.365247 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 09:08:56.365631 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":133,"spinlock_wait_cycles":10496,"thread_start_us":88,"threads_started":1}
I20260620 09:08:56.365855 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.347531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37021 (local address 127.0.98.3:33831)
0620 09:08:56.347851 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 09:08:56.347879 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 09:08:56.347912 (+ 33us) server_negotiation.cc:407] Connection header received
0620 09:08:56.348546 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:08:56.348552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:08:56.348838 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:08:56.349158 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:08:56.350395 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.351336 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.352497 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.352965 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.353168 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:56.360991 (+ 7823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:08:56.360997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:08:56.361012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:08:56.361088 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:08:56.363077 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363741 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:56.363746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:56.363750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:56.363861 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:08:56.364199 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:08:56.364207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:08:56.364209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:08:56.364617 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:08:56.364771 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 09:08:56.365361 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 09:08:56.365647 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":141,"threads_started":1}
I20260620 09:08:56.365895 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.347320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33831 (local address 127.0.98.1:37021)
0620 09:08:56.347594 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:56.347607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:08:56.347705 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:08:56.349137 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:08:56.349143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:08:56.349698 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:08:56.350192 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:56.350206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.351479 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.351483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:56.352370 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:08:56.352379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:08:56.353109 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:08:56.353113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:08:56.353318 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:08:56.358991 (+ 5673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:08:56.359011 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:08:56.360708 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:08:56.363233 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:56.363253 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:56.363594 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:56.363976 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:08:56.363978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:08:56.363981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:08:56.364090 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:08:56.364739 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:08:56.364746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:08:56.364985 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 09:08:56.365240 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 09:08:56.365702 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":323,"spinlock_wait_cycles":44032,"thread_start_us":52,"threads_started":1}
W20260620 09:08:56.368408 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
W20260620 09:08:56.368654 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
I20260620 09:08:56.368726 16959 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304; no voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785
I20260620 09:08:56.369021 17345 raft_consensus.cc:2726] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:08:56.555398 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.555017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:44661)
0620 09:08:56.555179 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:56.555279 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 09:08:56.716073 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.715643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:55027)
0620 09:08:56.715900 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:56.715960 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260620 09:08:56.931114 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:56.930745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:54311)
0620 09:08:56.930969 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:56.931021 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260620 09:08:57.556720 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:57.556243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:36279)
0620 09:08:57.556545 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:57.556603 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260620 09:08:57.717317 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:57.716845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:36415)
0620 09:08:57.717161 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:57.717221 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260620 09:08:57.932397 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:57.931954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:57993)
0620 09:08:57.932221 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:57.932285 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260620 09:08:58.160737 17355 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:08:58.160930 17355 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:08:58.161428 17355 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
W20260620 09:08:58.162424 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
W20260620 09:08:58.162533 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
I20260620 09:08:58.162670 16959 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304; no voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785
I20260620 09:08:58.162859 17355 raft_consensus.cc:2726] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:08:58.558004 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:58.557583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:41883)
0620 09:08:58.557842 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:58.557903 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260620 09:08:58.718420 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:58.718020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:39337)
0620 09:08:58.718272 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:58.718328 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260620 09:08:58.933708 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:58.933298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:57067)
0620 09:08:58.933551 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:58.933614 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260620 09:08:59.559580 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:59.559089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:44399)
0620 09:08:59.559417 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:59.559482 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":75,"threads_started":1}
I20260620 09:08:59.719610 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:59.719169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:60135)
0620 09:08:59.719445 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:59.719508 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260620 09:08:59.935040 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:08:59.934597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:47019)
0620 09:08:59.934848 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 09:08:59.934904 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260620 09:09:00.196107 17362 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:00.196233 17362 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:09:00.196683 17362 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
W20260620 09:09:00.197616 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
W20260620 09:09:00.197729 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
I20260620 09:09:00.197808 16959 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304; no voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785
I20260620 09:09:00.198071 17362 raft_consensus.cc:2726] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:09:00.560798 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:00.560376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:52427)
0620 09:09:00.560629 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:00.560687 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260620 09:09:00.720721 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:00.720293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:39193)
0620 09:09:00.720562 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:00.720615 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260620 09:09:00.936373 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:00.935910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:60101)
0620 09:09:00.936188 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:00.936248 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260620 09:09:01.562155 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:01.561685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:36149)
0620 09:09:01.561996 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:01.562058 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":118,"threads_started":1}
I20260620 09:09:01.721958 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:01.721519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:60507)
0620 09:09:01.721807 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:01.721865 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":69,"threads_started":1}
I20260620 09:09:01.937808 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:01.937277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:46823)
0620 09:09:01.937635 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:01.937700 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260620 09:09:02.006106 17369 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:02.006243 17369 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:09:02.006669 17369 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
W20260620 09:09:02.007570 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
W20260620 09:09:02.007730 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
I20260620 09:09:02.007797 16959 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304; no voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785
I20260620 09:09:02.007968 17369 raft_consensus.cc:2726] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:09:02.563365 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:02.562951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:34403)
0620 09:09:02.563193 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:02.563249 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260620 09:09:02.723174 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:02.722756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:35525)
0620 09:09:02.723014 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:02.723077 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260620 09:09:02.939155 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:02.938724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:41051)
0620 09:09:02.938992 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:02.939055 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260620 09:09:03.564670 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:03.564247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:32961)
0620 09:09:03.564513 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:03.564573 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260620 09:09:03.724511 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:03.724113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:50549)
0620 09:09:03.724354 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:03.724411 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260620 09:09:03.940415 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:03.940005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:52795)
0620 09:09:03.940268 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:03.940318 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260620 09:09:04.566049 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:04.565630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:35343)
0620 09:09:04.565896 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:04.565951 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260620 09:09:04.725903 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:04.725448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:54669)
0620 09:09:04.725739 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:04.725805 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260620 09:09:04.941561 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:04.941164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:45739)
0620 09:09:04.941409 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:04.941464 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260620 09:09:05.567199 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:05.566802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:44513)
0620 09:09:05.567044 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:05.567101 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260620 09:09:05.727090 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:05.726693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:44077)
0620 09:09:05.726936 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:05.726992 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260620 09:09:05.942713 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:05.942301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:36913)
0620 09:09:05.942548 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:05.942600 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260620 09:09:06.568535 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:06.568102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:47141)
0620 09:09:06.568351 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:06.568414 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260620 09:09:06.728502 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:06.728028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:33825)
0620 09:09:06.728347 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:06.728406 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260620 09:09:06.943856 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:06.943473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:40517)
0620 09:09:06.943711 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:06.943762 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260620 09:09:07.569803 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:07.569351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:49413)
0620 09:09:07.569636 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:07.569694 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260620 09:09:07.729878 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:07.729427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:36801)
0620 09:09:07.729715 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:07.729778 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260620 09:09:07.945004 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:07.944577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:59179)
0620 09:09:07.944839 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:07.944889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260620 09:09:08.014231 17388 raft_consensus.cc:482] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:08.014374 17388 raft_consensus.cc:504] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c10a16140024fe091049b2d2b672e3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34529 } } peers { permanent_uuid: "cf0874dbddb04923ad333adaade22785" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33831 } } peers { permanent_uuid: "92e284ba9e1045b0b1f7552ffd9fb304" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41639 } }
I20260620 09:09:08.014748 17388 leader_election.cc:288] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529), cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831)
W20260620 09:09:08.015597 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf0874dbddb04923ad333adaade22785 (127.0.98.3:33831): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
W20260620 09:09:08.015844 16959 leader_election.cc:341] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c10a16140024fe091049b2d2b672e3b (127.0.98.2:34529): Not found: Tablet not found: 020a34f51dbd4495a108fc542f412fb9
I20260620 09:09:08.015913 16959 leader_election.cc:302] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92e284ba9e1045b0b1f7552ffd9fb304; no voters: 9c10a16140024fe091049b2d2b672e3b, cf0874dbddb04923ad333adaade22785
I20260620 09:09:08.016105 17388 raft_consensus.cc:2726] T 020a34f51dbd4495a108fc542f412fb9 P 92e284ba9e1045b0b1f7552ffd9fb304 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 09:09:08.571007 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:08.570561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:44759)
0620 09:09:08.570807 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:08.570867 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260620 09:09:08.731071 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:08.730649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:58995)
0620 09:09:08.730897 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:08.730954 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260620 09:09:08.946238 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:08.945821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.3:55747)
0620 09:09:08.946079 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:08.946141 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
I20260620 09:09:09.572526 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:09.571973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.1:46407)
0620 09:09:09.572327 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:09.572422 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260620 09:09:09.732502 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:09.732050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.98.2:36719)
0620 09:09:09.732351 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:09.732413 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
I20260620 09:09:09.934150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 16902
I20260620 09:09:09.939879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17075
I20260620 09:09:09.945536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17209
2026-06-20T09:09:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59232 ms)
[----------] 30 tests from SecurityITest (240096 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:10 dist-test-slave-92t4 krb5kdc[17396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:10 dist-test-slave-92t4 krb5kdc[17396](info): set up 2 sockets
Jun 20 09:09:10 dist-test-slave-92t4 krb5kdc[17396](info): commencing operation
krb5kdc: starting...
W20260620 09:09:10.992877 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:11.114226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37423
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:11.229015 17409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:11.229394 17409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:11.229490 17409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:11.233604 17409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:11.233700 17409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:11.233736 17409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:11.233791 17409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:11.233842 17409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:11.239452 17409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:11.240623 17409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:11.241763 17409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:11.251287 17409 server_base.cc:1034] running on GCE node
W20260620 09:09:11.251243 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:11.251251 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:11.251240 17417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:11.252123 17409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:11.253346 17409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:11.254562 17409 hybrid_clock.cc:648] HybridClock initialized: now 1781946551254544 us; error 32 us; skew 500 ppm
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:11.257432 17409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:11.258363 17409 webserver.cc:460] Webserver started at http://127.0.98.62:41913/ using document root <none> and password file <none>
I20260620 09:09:11.258922 17409 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:11.259013 17409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:11.259240 17409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:11.261345 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "350be1ce44de4792b28139c39efa2193"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "1ca2b044a0eebbda77f7be6950fb57ba"
server_key_iv: "3a4e47f2a911d7e9b1ac8b2d086e5327"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.261917 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "350be1ce44de4792b28139c39efa2193"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "1ca2b044a0eebbda77f7be6950fb57ba"
server_key_iv: "3a4e47f2a911d7e9b1ac8b2d086e5327"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.265796 17409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:09:11.268282 17424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.269675 17409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:11.269778 17409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "350be1ce44de4792b28139c39efa2193"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "1ca2b044a0eebbda77f7be6950fb57ba"
server_key_iv: "3a4e47f2a911d7e9b1ac8b2d086e5327"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.269866 17409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:11.303412 17409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:11.306938 17409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:11.307112 17409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:11.315054 17409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37423
I20260620 09:09:11.315062 17476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37423 every 8 connection(s)
I20260620 09:09:11.316125 17409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:11.319049 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:11.321411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17409
I20260620 09:09:11.321552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:11.321866 392 external_mini_cluster.cc:1428] Setting key 36889a6e8ac491f05ddd94437ad17d90
I20260620 09:09:11.325556 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: Bootstrap starting.
I20260620 09:09:11.328008 17477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:11.328838 17477 log.cc:824] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:11.331166 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: No bootstrap required, opened a new log
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:11.335791 17477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } }
I20260620 09:09:11.336079 17477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:11.336172 17477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350be1ce44de4792b28139c39efa2193, State: Initialized, Role: FOLLOWER
I20260620 09:09:11.336670 17477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } }
I20260620 09:09:11.336858 17477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:11.336925 17477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:11.337014 17477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:11.337486 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:11.323454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.62:37423)
0620 09:09:11.323917 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:11.323933 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:11.323971 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:09:11.324656 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:11.324673 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:11.325107 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:11.325426 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:11.326312 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.327426 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.328589 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.329239 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.329423 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:11.332183 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:11.332189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:11.332204 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:11.332233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:11.334342 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.334788 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.334794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.334801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.334950 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.335266 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.335269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.335271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.335749 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:11.335950 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:11.336150 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 09:09:11.336457 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":159,"threads_started":1}
I20260620 09:09:11.338097 17477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } }
I20260620 09:09:11.338449 17477 leader_election.cc:302] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350be1ce44de4792b28139c39efa2193; no voters:
I20260620 09:09:11.338745 17477 leader_election.cc:288] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:11.338850 17482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:11.339054 17482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [term 1 LEADER]: Becoming Leader. State: Replica: 350be1ce44de4792b28139c39efa2193, State: Running, Role: LEADER
I20260620 09:09:11.339411 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } }
I20260620 09:09:11.340031 17477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:11.340886 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350be1ce44de4792b28139c39efa2193. Latest consensus state: current_term: 1 leader_uuid: "350be1ce44de4792b28139c39efa2193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } } }
I20260620 09:09:11.341111 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:11.341342 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350be1ce44de4792b28139c39efa2193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350be1ce44de4792b28139c39efa2193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37423 } } }
I20260620 09:09:11.341454 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:11.341634 17491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:11.344599 17491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:11.350167 17491 catalog_manager.cc:1347] Generated new cluster ID: 58eafc16ef7f441bb5074f71c6b6326d
I20260620 09:09:11.350245 17491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:11.369539 17491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:11.370533 17491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:11.388762 17491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: Generated new TSK 0
I20260620 09:09:11.389598 17491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:09:11.422036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:11.536098 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:11.536368 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:11.536412 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:11.540855 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:11.540920 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:11.541021 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:11.546540 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:11.547698 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:11.548774 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:11.558773 17511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:11.558774 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:11.558956 17503 server_base.cc:1034] running on GCE node
W20260620 09:09:11.559072 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:11.559527 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:11.560214 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:11.561398 17503 hybrid_clock.cc:648] HybridClock initialized: now 1781946551561377 us; error 37 us; skew 500 ppm
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:11.564239 17503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:09:11.565265 17503 webserver.cc:460] Webserver started at http://127.0.98.1:32855/ using document root <none> and password file <none>
I20260620 09:09:11.565857 17503 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:11.565949 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:11.566188 17503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:11.568308 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "6783916397d84e63b9fff6dbef5806b2"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "5a3d0d765f4ab773c621e983887308ad"
server_key_iv: "07282a11d3a4d77a796761a0b10584be"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.568892 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6783916397d84e63b9fff6dbef5806b2"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "5a3d0d765f4ab773c621e983887308ad"
server_key_iv: "07282a11d3a4d77a796761a0b10584be"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.572674 17503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:09:11.575225 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.576465 17503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:11.576578 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "6783916397d84e63b9fff6dbef5806b2"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "5a3d0d765f4ab773c621e983887308ad"
server_key_iv: "07282a11d3a4d77a796761a0b10584be"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.576665 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:11.594929 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:11.598587 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:11.598753 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:11.599301 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:11.600109 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:11.600158 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.600199 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:11.600214 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.610402 17503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44623
I20260620 09:09:11.610430 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44623 every 8 connection(s)
I20260620 09:09:11.611371 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:11.619012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17503
I20260620 09:09:11.619128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:11.619460 392 external_mini_cluster.cc:1428] Setting key 7017275c75609d59ec0bc3a9a2592287
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:11.625711 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:11.613389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42579 (local address 127.0.98.62:37423)
0620 09:09:11.613530 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:11.613533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:11.614396 (+ 863us) server_negotiation.cc:407] Connection header received
0620 09:09:11.615340 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:11.615343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:11.615406 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:11.615509 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:11.616739 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.617350 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.618542 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.618799 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.618921 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:11.621947 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:11.621951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:11.621953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:11.621976 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:11.623457 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.624041 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.624046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.624048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.624135 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.624492 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.624495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.624497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.624683 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:11.624792 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:11.625400 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 09:09:11.625542 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:11.626552 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:11.613790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37423 (local address 127.0.98.1:42579)
0620 09:09:11.614247 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:11.614291 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:11.615121 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:11.615712 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:11.615719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:11.616089 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:11.616575 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:11.616589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.617486 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.617490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:11.618406 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:11.618415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.618965 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.618970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:11.619229 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:11.619921 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:11.619939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:11.621764 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:11.623600 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:11.623606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:11.623623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:11.623927 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:11.624231 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:11.624235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:11.624237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:11.624389 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:11.624791 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:11.624798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:11.625145 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 09:09:11.625359 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 09:09:11.625705 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260620 09:09:11.627846 17632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37423
I20260620 09:09:11.628139 17632 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:11.628750 17632 heartbeater.cc:511] Master 127.0.98.62:37423 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:09:11.630415 17441 ts_manager.cc:194] Registered new tserver with Master: 6783916397d84e63b9fff6dbef5806b2 (127.0.98.1:44623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:09:11.631906 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:09:11.647666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:11.759652 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:11.759938 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:11.759982 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:11.764374 17637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:11.764444 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:11.764550 17637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:11.770038 17637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:11.771164 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:11.772239 17637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:11.781891 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:11.781895 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:11.781852 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:11.782217 17637 server_base.cc:1034] running on GCE node
I20260620 09:09:11.782618 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:11.783272 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:11.784463 17637 hybrid_clock.cc:648] HybridClock initialized: now 1781946551784442 us; error 31 us; skew 500 ppm
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:11.787307 17637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:09:11.788232 17637 webserver.cc:460] Webserver started at http://127.0.98.2:33093/ using document root <none> and password file <none>
I20260620 09:09:11.788815 17637 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:11.788908 17637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:11.789209 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:11.791330 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "5aef3b3ce756404b86e271cf6ae3e3cd"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "d1f2b07723d5fcf6dc40e66a23e7dea8"
server_key_iv: "de98849e76fcba00f23f8151584c809a"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.791908 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5aef3b3ce756404b86e271cf6ae3e3cd"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "d1f2b07723d5fcf6dc40e66a23e7dea8"
server_key_iv: "de98849e76fcba00f23f8151584c809a"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.795858 17637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 09:09:11.798532 17652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.799758 17637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:11.799904 17637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "5aef3b3ce756404b86e271cf6ae3e3cd"
format_stamp: "Formatted at 2026-06-20 09:09:11 on dist-test-slave-92t4"
server_key: "d1f2b07723d5fcf6dc40e66a23e7dea8"
server_key_iv: "de98849e76fcba00f23f8151584c809a"
server_key_version: "encryptionkey@0"
I20260620 09:09:11.800031 17637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:11.815201 17637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:11.818482 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:11.818706 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:11.819336 17637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:11.820218 17637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:11.820291 17637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.820365 17637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:11.820408 17637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:11.829659 17637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38701
I20260620 09:09:11.829682 17765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38701 every 8 connection(s)
I20260620 09:09:11.830652 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:11.834110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17637
I20260620 09:09:11.834246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:09:11.834578 392 external_mini_cluster.cc:1428] Setting key fbd89a5d09ffd6dcf66acc4009cdf482
Jun 20 09:09:11 dist-test-slave-92t4 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:09:11.846645 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:11.832523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51231 (local address 127.0.98.62:37423)
0620 09:09:11.832694 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:11.832697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:11.833710 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 09:09:11.835211 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:11.835214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:11.835269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:11.835353 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:11.837133 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.837993 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.839198 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.839419 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.839504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:11.842956 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:11.842959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:11.842961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:11.842986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:11.844428 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.845100 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.845104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.845107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.845208 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:11.845586 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:11.845598 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:11.845599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:11.845789 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:11.845869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:11.846399 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 09:09:11.846500 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 09:09:11.847517 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:11.832907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37423 (local address 127.0.98.2:51231)
0620 09:09:11.833439 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:11.833485 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:11.834788 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:11.835617 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:11.835628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:11.836174 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:11.836882 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:11.836904 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.838174 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.838177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:11.839084 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:11.839093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:11.839597 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:11.839604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:11.839843 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:11.840520 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:11.840546 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:11.842768 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:11.844555 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:11.844561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:11.844586 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:11.844928 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:11.845342 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:11.845348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:11.845352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:11.845495 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:11.845885 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:11.845890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:11.846175 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 09:09:11.846339 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 09:09:11.846614 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":181,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:09:11.848883 17766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37423
I20260620 09:09:11.849189 17766 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:11.849685 17766 heartbeater.cc:511] Master 127.0.98.62:37423 requested a full tablet report, sending...
I20260620 09:09:11.850750 17441 ts_manager.cc:194] Registered new tserver with Master: 5aef3b3ce756404b86e271cf6ae3e3cd (127.0.98.2:38701)
I20260620 09:09:11.851332 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:09:11.863528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:11.977000 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:11.977324 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:11.977406 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:11.981966 17771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:11.982057 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:11.982147 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:11.987637 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:11.988839 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:11.989995 17771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:12.001765 17779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:12.001914 17771 server_base.cc:1034] running on GCE node
W20260620 09:09:12.001747 17776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:12.001747 17777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:12.002671 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:12.003408 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:12.004619 17771 hybrid_clock.cc:648] HybridClock initialized: now 1781946552004605 us; error 27 us; skew 500 ppm
Jun 20 09:09:12 dist-test-slave-92t4 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:12.007560 17771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:09:12.008500 17771 webserver.cc:460] Webserver started at http://127.0.98.3:32829/ using document root <none> and password file <none>
I20260620 09:09:12.009230 17771 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:12.009325 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:12.009552 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:12.011690 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d7b2b5e34314c62a1e80eeac497804d"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "9b8b23e57e3f9ef809d187575dc394ac"
server_key_iv: "b0d33e655e6b571e5a90e238fc758c5b"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.012272 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d7b2b5e34314c62a1e80eeac497804d"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "9b8b23e57e3f9ef809d187575dc394ac"
server_key_iv: "b0d33e655e6b571e5a90e238fc758c5b"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.016233 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:12.018860 17786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.020228 17771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:12.020381 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "1d7b2b5e34314c62a1e80eeac497804d"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "9b8b23e57e3f9ef809d187575dc394ac"
server_key_iv: "b0d33e655e6b571e5a90e238fc758c5b"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.020504 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:12.054633 17771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:12.058374 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:12.058611 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:12.059247 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:12.060206 17771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:12.060276 17771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.060349 17771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:12.060401 17771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.069692 17771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36763
I20260620 09:09:12.069715 17899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36763 every 8 connection(s)
I20260620 09:09:12.070686 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:12.080287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17771
I20260620 09:09:12.080454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:12.080824 392 external_mini_cluster.cc:1428] Setting key b1a109cf5415b4d223fbad7d77e9be86
Jun 20 09:09:12 dist-test-slave-92t4 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:12.086300 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.072764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47691 (local address 127.0.98.62:37423)
0620 09:09:12.072935 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.072940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.074244 (+ 1304us) server_negotiation.cc:407] Connection header received
0620 09:09:12.075262 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.075265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.075314 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.075391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:12.076623 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.077241 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.078568 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.078874 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.078983 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.082356 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:12.082360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:12.082363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:12.082394 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:12.084015 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:12.084514 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:12.084517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:12.084518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:12.084595 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:12.085199 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:12.085202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:12.085203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:12.085392 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:12.085483 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.086060 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.086176 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:09:12.087227 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.073243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37423 (local address 127.0.98.3:47691)
0620 09:09:12.074084 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.074133 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.075054 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.075570 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.075576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.075950 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:12.076473 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.076488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.077381 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.077385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.078435 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.078445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.079023 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.079026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.079201 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.080066 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:12.080091 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:12.082173 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:12.084143 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:12.084148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:12.084161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:12.084423 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:12.084716 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:12.084718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:12.084720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:12.084868 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:12.085538 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:12.085544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:12.085802 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.085969 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.086260 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":125,"threads_started":1}
I20260620 09:09:12.088546 17900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37423
I20260620 09:09:12.088841 17900 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:12.089483 17900 heartbeater.cc:511] Master 127.0.98.62:37423 requested a full tablet report, sending...
I20260620 09:09:12.090641 17441 ts_manager.cc:194] Registered new tserver with Master: 1d7b2b5e34314c62a1e80eeac497804d (127.0.98.3:36763)
I20260620 09:09:12.091274 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47691
I20260620 09:09:12.095283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:12.105240 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.097405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.0.98.62:37423)
0620 09:09:12.097526 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.097529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.097698 (+ 169us) server_negotiation.cc:407] Connection header received
0620 09:09:12.097781 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.097783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.097825 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.097895 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:12.098639 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.099158 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.100004 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.100227 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.100307 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.101974 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:12.101977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:12.101979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:12.102005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:12.103545 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:12.103935 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:12.103938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:12.103939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:12.104015 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:12.104360 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:12.104362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:12.104364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:12.104587 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:12.104693 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.104964 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.105118 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:09:12.108191 17441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:12.111428 17441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:12.127424 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.120873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38701 (local address 127.0.0.1:55724)
0620 09:09:12.121761 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.121815 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.121951 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.122767 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.122773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.122803 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:12.122993 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.123001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.124088 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.124091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.125818 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.125829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.126819 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.126822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.126937 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.126975 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.127118 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.127243 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":25,"spinlock_wait_cycles":8576,"thread_start_us":93,"threads_started":1}
I20260620 09:09:12.127604 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.121412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.98.2:38701)
0620 09:09:12.122301 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.122314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.122338 (+ 24us) server_negotiation.cc:407] Connection header received
0620 09:09:12.122414 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.122423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.122612 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.122765 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:12.123125 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.123967 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.125949 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.126690 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.126822 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.126997 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.127149 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.127428 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":92,"threads_started":1}
I20260620 09:09:12.127902 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.120873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44623 (local address 127.0.0.1:43356)
0620 09:09:12.121929 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.121941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.122015 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.122766 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.122775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.122813 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:12.122991 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.122999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.125269 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.125274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.126267 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.126277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.127413 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.127416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.127577 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.127613 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.127679 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.127741 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":302,"spinlock_wait_cycles":6656,"thread_start_us":100,"threads_started":1}
I20260620 09:09:12.128468 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.121100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36763 (local address 127.0.0.1:47350)
0620 09:09:12.121760 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.121817 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.121952 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.123496 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.123499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.123516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:12.123615 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.123625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.125013 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.125016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.126136 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.126144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.127413 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.127416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.127940 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.128195 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.128269 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.128336 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":31,"spinlock_wait_cycles":2816,"thread_start_us":103,"threads_started":1}
I20260620 09:09:12.128473 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.121058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.98.1:44623)
0620 09:09:12.122300 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.122316 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.122338 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:09:12.122434 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.122440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.122618 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.122772 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:12.124106 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.125122 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.126389 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.127142 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.127232 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.127352 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.128247 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.128344 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":62,"threads_started":1}
I20260620 09:09:12.129070 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.121520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.98.3:36763)
0620 09:09:12.123117 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.123133 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.123153 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:12.123219 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.123222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.123346 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.123497 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:12.123741 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.124823 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.126249 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.127087 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.127209 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.127347 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.128797 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.128895 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":101,"threads_started":1}
I20260620 09:09:12.130786 17700 tablet_service.cc:1467] Processing CreateTablet for tablet 2b886a7d482a4602980e8ad745c43415 (DEFAULT_TABLE table=test-table [id=0d35192215fc48f795b1dbe2c2e12f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:12.131235 17566 tablet_service.cc:1467] Processing CreateTablet for tablet 2b886a7d482a4602980e8ad745c43415 (DEFAULT_TABLE table=test-table [id=0d35192215fc48f795b1dbe2c2e12f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:12.132002 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b886a7d482a4602980e8ad745c43415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:12.132290 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b886a7d482a4602980e8ad745c43415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:12.137331 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 2b886a7d482a4602980e8ad745c43415 (DEFAULT_TABLE table=test-table [id=0d35192215fc48f795b1dbe2c2e12f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:12.138474 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b886a7d482a4602980e8ad745c43415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:12.139751 17917 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Bootstrap starting.
I20260620 09:09:12.141105 17918 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Bootstrap starting.
I20260620 09:09:12.142287 17917 tablet_bootstrap.cc:654] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:12.143271 17917 log.cc:824] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:12.143360 17918 tablet_bootstrap.cc:654] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:12.144286 17918 log.cc:824] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:12.146052 17918 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: No bootstrap required, opened a new log
I20260620 09:09:12.146068 17917 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: No bootstrap required, opened a new log
I20260620 09:09:12.146250 17917 ts_tablet_manager.cc:1397] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 09:09:12.146251 17918 ts_tablet_manager.cc:1397] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:09:12.146271 17921 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Bootstrap starting.
I20260620 09:09:12.148779 17921 tablet_bootstrap.cc:654] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:12.150090 17921 log.cc:824] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:12.150358 17917 raft_consensus.cc:348] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.150650 17917 raft_consensus.cc:374] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:12.150724 17917 raft_consensus.cc:729] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6783916397d84e63b9fff6dbef5806b2, State: Initialized, Role: FOLLOWER
I20260620 09:09:12.150645 17918 raft_consensus.cc:348] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.150950 17918 raft_consensus.cc:374] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:12.151209 17918 raft_consensus.cc:729] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aef3b3ce756404b86e271cf6ae3e3cd, State: Initialized, Role: FOLLOWER
I20260620 09:09:12.151222 17917 consensus_queue.cc:260] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.151669 17918 consensus_queue.cc:260] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.152225 17921 tablet_bootstrap.cc:492] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: No bootstrap required, opened a new log
I20260620 09:09:12.152266 17917 ts_tablet_manager.cc:1428] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:09:12.152415 17921 ts_tablet_manager.cc:1397] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:09:12.152550 17632 heartbeater.cc:503] Master 127.0.98.62:37423 was elected leader, sending a full tablet report...
I20260620 09:09:12.152567 17918 ts_tablet_manager.cc:1428] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:09:12.152915 17766 heartbeater.cc:503] Master 127.0.98.62:37423 was elected leader, sending a full tablet report...
I20260620 09:09:12.156308 17921 raft_consensus.cc:348] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.156617 17921 raft_consensus.cc:374] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:12.156695 17921 raft_consensus.cc:729] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7b2b5e34314c62a1e80eeac497804d, State: Initialized, Role: FOLLOWER
I20260620 09:09:12.157229 17921 consensus_queue.cc:260] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.157900 17900 heartbeater.cc:503] Master 127.0.98.62:37423 was elected leader, sending a full tablet report...
I20260620 09:09:12.158288 17921 ts_tablet_manager.cc:1428] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:12.320019 17924 raft_consensus.cc:482] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:12.320253 17924 raft_consensus.cc:504] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.321614 17924 leader_election.cc:288] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6783916397d84e63b9fff6dbef5806b2 (127.0.98.1:44623), 1d7b2b5e34314c62a1e80eeac497804d (127.0.98.3:36763)
W20260620 09:09:12.321679 17901 tablet.cc:2369] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:12.326370 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.322178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46023 (local address 127.0.98.1:44623)
0620 09:09:12.322287 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.322292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.322317 (+ 25us) server_negotiation.cc:407] Connection header received
0620 09:09:12.322381 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.322383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.322452 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.322548 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:12.322975 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.323664 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.325177 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.325805 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.325920 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.325987 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.326174 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.326242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 09:09:12.326370 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.321832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44623 (local address 127.0.98.2:46023)
0620 09:09:12.321961 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.321980 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.322089 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.322686 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.322689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.322710 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:12.322831 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.322840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.323810 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.323812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.324992 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.325003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.325922 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.325925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.326037 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.326059 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.326158 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.326236 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:09:12.327033 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b886a7d482a4602980e8ad745c43415" candidate_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6783916397d84e63b9fff6dbef5806b2" is_pre_election: true
I20260620 09:09:12.327131 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.322588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55617 (local address 127.0.98.3:36763)
0620 09:09:12.322755 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.322760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.322904 (+ 144us) server_negotiation.cc:407] Connection header received
0620 09:09:12.323021 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.323023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.323074 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.323159 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:12.323427 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.324215 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.325712 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.326314 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.326412 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.326479 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.326936 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.327014 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 09:09:12.327358 17586 raft_consensus.cc:2454] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aef3b3ce756404b86e271cf6ae3e3cd in term 0.
I20260620 09:09:12.327579 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.322467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36763 (local address 127.0.98.2:55617)
0620 09:09:12.322804 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.322819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.322919 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.323186 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.323189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.323207 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:12.323317 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.323322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.324350 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.324353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.325594 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.325603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.326688 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.326692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.326792 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.326815 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.327328 (+ 513us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.327402 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":85,"threads_started":1}
I20260620 09:09:12.327883 17653 leader_election.cc:302] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aef3b3ce756404b86e271cf6ae3e3cd, 6783916397d84e63b9fff6dbef5806b2; no voters:
I20260620 09:09:12.328173 17924 raft_consensus.cc:2781] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:12.328256 17924 raft_consensus.cc:482] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:12.328289 17924 raft_consensus.cc:3037] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:12.328222 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b886a7d482a4602980e8ad745c43415" candidate_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7b2b5e34314c62a1e80eeac497804d" is_pre_election: true
I20260620 09:09:12.328485 17854 raft_consensus.cc:2454] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aef3b3ce756404b86e271cf6ae3e3cd in term 0.
I20260620 09:09:12.329551 17924 raft_consensus.cc:504] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.329962 17924 leader_election.cc:288] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [CANDIDATE]: Term 1 election: Requested vote from peers 6783916397d84e63b9fff6dbef5806b2 (127.0.98.1:44623), 1d7b2b5e34314c62a1e80eeac497804d (127.0.98.3:36763)
I20260620 09:09:12.330282 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b886a7d482a4602980e8ad745c43415" candidate_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6783916397d84e63b9fff6dbef5806b2"
I20260620 09:09:12.330417 17586 raft_consensus.cc:3037] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:12.330351 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b886a7d482a4602980e8ad745c43415" candidate_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d7b2b5e34314c62a1e80eeac497804d"
I20260620 09:09:12.330610 17854 raft_consensus.cc:3037] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:12.331513 17586 raft_consensus.cc:2454] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aef3b3ce756404b86e271cf6ae3e3cd in term 1.
I20260620 09:09:12.331694 17854 raft_consensus.cc:2454] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aef3b3ce756404b86e271cf6ae3e3cd in term 1.
W20260620 09:09:12.331812 17767 tablet.cc:2369] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:12.331810 17653 leader_election.cc:302] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aef3b3ce756404b86e271cf6ae3e3cd, 6783916397d84e63b9fff6dbef5806b2; no voters:
I20260620 09:09:12.332037 17924 raft_consensus.cc:2781] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:12.332249 17924 raft_consensus.cc:686] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 1 LEADER]: Becoming Leader. State: Replica: 5aef3b3ce756404b86e271cf6ae3e3cd, State: Running, Role: LEADER
I20260620 09:09:12.332612 17924 consensus_queue.cc:237] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } }
I20260620 09:09:12.336006 17438 catalog_manager.cc:5495] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aef3b3ce756404b86e271cf6ae3e3cd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aef3b3ce756404b86e271cf6ae3e3cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 } health_report { overall_health: UNKNOWN } } }
W20260620 09:09:12.362823 17633 tablet.cc:2369] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:12.365664 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.361623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.0.98.2:38701)
0620 09:09:12.361782 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.361787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.361816 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:09:12.361973 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.361976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.362033 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.362158 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:12.362409 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.363115 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.364040 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.364257 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.364344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.364660 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:12.365214 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:12.365322 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.365433 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.365488 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:09:12.371860 17586 raft_consensus.cc:1261] T 2b886a7d482a4602980e8ad745c43415 P 6783916397d84e63b9fff6dbef5806b2 [term 1 FOLLOWER]: Refusing update from remote peer 5aef3b3ce756404b86e271cf6ae3e3cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:12.371860 17854 raft_consensus.cc:1261] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 1 FOLLOWER]: Refusing update from remote peer 5aef3b3ce756404b86e271cf6ae3e3cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:12.372627 17927 consensus_queue.cc:1035] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6783916397d84e63b9fff6dbef5806b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:12.372826 17924 consensus_queue.cc:1035] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7b2b5e34314c62a1e80eeac497804d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:12.380513 17932 mvcc.cc:205] Tried to move back new op lower bound from 7298853078507732992 to 7298853078357270528. Current Snapshot: MvccSnapshot[applied={T|T < 7298853078507732992}]
I20260620 09:09:12.383437 17931 mvcc.cc:205] Tried to move back new op lower bound from 7298853078507732992 to 7298853078357270528. Current Snapshot: MvccSnapshot[applied={T|T < 7298853078507732992}]
I20260620 09:09:12.389034 17441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:12.389246 17441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:12.391853 17441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350be1ce44de4792b28139c39efa2193: Sending DeleteTablet for 3 replicas of tablet 2b886a7d482a4602980e8ad745c43415
I20260620 09:09:12.392700 17700 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b886a7d482a4602980e8ad745c43415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55724
I20260620 09:09:12.392702 17566 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b886a7d482a4602980e8ad745c43415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260620 09:09:12.392848 17834 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b886a7d482a4602980e8ad745c43415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47350
I20260620 09:09:12.393386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17503
I20260620 09:09:12.393409 17940 tablet_replica.cc:330] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: stopping tablet replica
I20260620 09:09:12.393750 17940 raft_consensus.cc:2229] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:12.394289 17940 raft_consensus.cc:2258] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:12.393397 17941 tablet_replica.cc:330] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: stopping tablet replica
I20260620 09:09:12.395176 17941 raft_consensus.cc:2229] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:12.395608 17941 raft_consensus.cc:2258] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:12.397938 17941 ts_tablet_manager.cc:1905] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:12.397984 17940 ts_tablet_manager.cc:1905] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:12.400774 17941 ts_tablet_manager.cc:1918] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:12.400774 17940 ts_tablet_manager.cc:1918] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:12.400887 17941 log.cc:1196] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/2b886a7d482a4602980e8ad745c43415
I20260620 09:09:12.401055 17940 log.cc:1196] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/2b886a7d482a4602980e8ad745c43415
I20260620 09:09:12.401285 17941 ts_tablet_manager.cc:1939] T 2b886a7d482a4602980e8ad745c43415 P 1d7b2b5e34314c62a1e80eeac497804d: Deleting consensus metadata
I20260620 09:09:12.401516 17940 ts_tablet_manager.cc:1939] T 2b886a7d482a4602980e8ad745c43415 P 5aef3b3ce756404b86e271cf6ae3e3cd: Deleting consensus metadata
W20260620 09:09:12.402174 17425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44623 (error 108)
I20260620 09:09:12.402696 17428 catalog_manager.cc:4841] TS 1d7b2b5e34314c62a1e80eeac497804d (127.0.98.3:36763): tablet 2b886a7d482a4602980e8ad745c43415 (table test-table [id=0d35192215fc48f795b1dbe2c2e12f93]) successfully deleted
I20260620 09:09:12.402745 17427 catalog_manager.cc:4841] TS 5aef3b3ce756404b86e271cf6ae3e3cd (127.0.98.2:38701): tablet 2b886a7d482a4602980e8ad745c43415 (table test-table [id=0d35192215fc48f795b1dbe2c2e12f93]) successfully deleted
I20260620 09:09:12.403108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17637
I20260620 09:09:12.404878 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.404533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44623 (local address 127.0.0.1:43370)
0620 09:09:12.404683 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.404766 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 09:09:12.405139 17425 catalog_manager.cc:4584] TS 6783916397d84e63b9fff6dbef5806b2 (127.0.98.1:44623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b886a7d482a4602980e8ad745c43415: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44623: connect: Connection refused (error 111)
I20260620 09:09:12.408484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17771
I20260620 09:09:12.414263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17409
2026-06-20T09:09:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2468 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T09:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:12Z Disabled control of system clock
I20260620 09:09:12.445286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33313
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:12.559201 17946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:12.559523 17946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:12.559561 17946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:12.563678 17946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:12.563752 17946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:12.563769 17946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:12.563786 17946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:12.563802 17946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:12.568712 17946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:12.569955 17946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:12.571429 17946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:12.580404 17951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:12.580410 17954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:12.580390 17952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:12.580690 17946 server_base.cc:1034] running on GCE node
I20260620 09:09:12.581491 17946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:12.582721 17946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:12.583925 17946 hybrid_clock.cc:648] HybridClock initialized: now 1781946552583909 us; error 31 us; skew 500 ppm
I20260620 09:09:12.586040 17946 webserver.cc:460] Webserver started at http://127.0.98.62:39459/ using document root <none> and password file <none>
I20260620 09:09:12.586679 17946 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:12.586776 17946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:12.587029 17946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:12.589354 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "89e5c1c418e94353a62915539489037e"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "be058f01088d4766c352f5c0bc2a300b"
server_key_iv: "0a03d8aef153e77c6f3507e3a8348cff"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.589984 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "89e5c1c418e94353a62915539489037e"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "be058f01088d4766c352f5c0bc2a300b"
server_key_iv: "0a03d8aef153e77c6f3507e3a8348cff"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.594312 17946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:09:12.597295 17960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.598811 17946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:12.599035 17946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "89e5c1c418e94353a62915539489037e"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "be058f01088d4766c352f5c0bc2a300b"
server_key_iv: "0a03d8aef153e77c6f3507e3a8348cff"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.599186 17946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:12.614681 17946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:12.615516 17946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:12.615746 17946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:12.623360 17946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33313
I20260620 09:09:12.623382 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33313 every 8 connection(s)
I20260620 09:09:12.624480 17946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:12.627948 18013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:12.631458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 17946
I20260620 09:09:12.631614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:12.631884 392 external_mini_cluster.cc:1428] Setting key 942fa52b22a76d4ce978dfea96001a21
I20260620 09:09:12.634580 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: Bootstrap starting.
I20260620 09:09:12.637264 18013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:12.638084 18013 log.cc:824] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:12.640350 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: No bootstrap required, opened a new log
I20260620 09:09:12.642407 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.633543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.98.62:33313)
0620 09:09:12.634336 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.634353 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.634395 (+ 42us) server_negotiation.cc:407] Connection header received
0620 09:09:12.635190 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.635211 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.635701 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.636136 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:12.636635 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.637806 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.638798 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.639313 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.639518 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.640497 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:12.640504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:12.640523 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:12.640944 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:12.641004 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:12.641026 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:12.641185 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.641354 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.641652 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":165,"threads_started":1}
I20260620 09:09:12.644229 18013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260620 09:09:12.644482 18013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:12.644548 18013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e5c1c418e94353a62915539489037e, State: Initialized, Role: FOLLOWER
I20260620 09:09:12.645030 18013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260620 09:09:12.645231 18013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:12.645318 18013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:12.645438 18013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:12.646397 18013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260620 09:09:12.646749 18013 leader_election.cc:302] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e5c1c418e94353a62915539489037e; no voters:
I20260620 09:09:12.647086 18013 leader_election.cc:288] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:12.647261 18018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:12.647519 18018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [term 1 LEADER]: Becoming Leader. State: Replica: 89e5c1c418e94353a62915539489037e, State: Running, Role: LEADER
I20260620 09:09:12.647843 18018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260620 09:09:12.648346 18013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:12.649526 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e5c1c418e94353a62915539489037e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } } }
I20260620 09:09:12.649540 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e5c1c418e94353a62915539489037e. Latest consensus state: current_term: 1 leader_uuid: "89e5c1c418e94353a62915539489037e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e5c1c418e94353a62915539489037e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } } }
I20260620 09:09:12.649672 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:12.649672 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:12.650112 18027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:12.652878 18027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:12.658455 18027 catalog_manager.cc:1347] Generated new cluster ID: f25afa30747048ee95a79f7e46579eb7
I20260620 09:09:12.658536 18027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:12.672758 18027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:12.673799 18027 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:12.682574 18027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: Generated new TSK 0
I20260620 09:09:12.683543 18027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:12.694101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--builtin_ntp_servers=127.0.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:12.812049 18037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:12.812350 18037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:12.812390 18037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:12.816584 18037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:12.816654 18037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:12.816713 18037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:12.821755 18037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:12.822876 18037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:12.824292 18037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:12.834321 18045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:12.834321 18042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:12.834342 18043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:12.834744 18037 server_base.cc:1034] running on GCE node
I20260620 09:09:12.835217 18037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:12.835979 18037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:12.837126 18037 hybrid_clock.cc:648] HybridClock initialized: now 1781946552837069 us; error 55 us; skew 500 ppm
I20260620 09:09:12.839048 18037 webserver.cc:460] Webserver started at http://127.0.98.1:45627/ using document root <none> and password file <none>
I20260620 09:09:12.839665 18037 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:12.839756 18037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:12.839977 18037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:12.842247 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "0db824b506634a2e994ba54502cbcea8"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "3b8ff5ef838673ea33f32be16b2c86dc"
server_key_iv: "11a768496f2d39592481aa9fe0a9b703"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.842844 18037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0db824b506634a2e994ba54502cbcea8"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "3b8ff5ef838673ea33f32be16b2c86dc"
server_key_iv: "11a768496f2d39592481aa9fe0a9b703"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.846694 18037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:12.849362 18051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.850720 18037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:09:12.850860 18037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "0db824b506634a2e994ba54502cbcea8"
format_stamp: "Formatted at 2026-06-20 09:09:12 on dist-test-slave-92t4"
server_key: "3b8ff5ef838673ea33f32be16b2c86dc"
server_key_iv: "11a768496f2d39592481aa9fe0a9b703"
server_key_version: "encryptionkey@0"
I20260620 09:09:12.850981 18037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:12.868221 18037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:12.868968 18037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:12.869225 18037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:12.869854 18037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:12.870836 18037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:12.870911 18037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.870986 18037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:12.871044 18037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:12.880172 18037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45647
I20260620 09:09:12.880195 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45647 every 8 connection(s)
I20260620 09:09:12.881258 18037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:12.890724 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.883130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44327 (local address 127.0.98.62:33313)
0620 09:09:12.883289 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:12.883294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:12.884457 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 09:09:12.885863 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:12.885866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:12.885915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:12.885993 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:12.886904 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.887437 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.888603 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.888838 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.888934 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.890175 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:12.890178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:12.890179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:12.890193 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:12.890211 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:12.890219 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:12.890281 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:12.890542 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:09:12.890602 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:09:12.890998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18037
I20260620 09:09:12.891117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:12.891512 392 external_mini_cluster.cc:1428] Setting key 11a5dfc5a9ac59c019d901cb4106acf6
I20260620 09:09:12.892009 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:12.883613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.1:44327)
0620 09:09:12.884298 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:12.884340 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:12.885651 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:12.886163 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:12.886170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:12.886258 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:12.886777 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.886789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.887543 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.887546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.888464 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:12.888473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:12.888980 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:12.888983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:12.889225 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:12.889916 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:12.889918 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:12.889962 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:12.890029 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:12.890324 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:12.890332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:12.890338 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:09:12.890477 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 09:09:12.890788 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260620 09:09:12.893411 18165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260620 09:09:12.893668 18165 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:12.893699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--builtin_ntp_servers=127.0.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:12.894325 18165 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260620 09:09:12.895937 17976 ts_manager.cc:194] Registered new tserver with Master: 0db824b506634a2e994ba54502cbcea8 (127.0.98.1:45647)
I20260620 09:09:12.897182 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44327
W20260620 09:09:13.008631 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:13.008934 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:13.008975 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:13.013128 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:13.013192 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:13.013252 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:13.018101 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:13.019325 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:13.020778 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:13.030652 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:13.030645 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:13.030903 18168 server_base.cc:1034] running on GCE node
W20260620 09:09:13.030645 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:13.031355 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:13.032037 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:13.033253 18168 hybrid_clock.cc:648] HybridClock initialized: now 1781946553033215 us; error 52 us; skew 500 ppm
I20260620 09:09:13.035368 18168 webserver.cc:460] Webserver started at http://127.0.98.2:33745/ using document root <none> and password file <none>
I20260620 09:09:13.036144 18168 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:13.036249 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:13.036513 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:13.038903 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "a5bc576a944c4c24bbc60db19ec07d4a"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "c4193d84037959231e981c81a766291b"
server_key_iv: "8f55fb00cdc9cd479e0800f7e60e60db"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.039522 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a5bc576a944c4c24bbc60db19ec07d4a"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "c4193d84037959231e981c81a766291b"
server_key_iv: "8f55fb00cdc9cd479e0800f7e60e60db"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.043596 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:13.046298 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.047564 18168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:13.047705 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "a5bc576a944c4c24bbc60db19ec07d4a"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "c4193d84037959231e981c81a766291b"
server_key_iv: "8f55fb00cdc9cd479e0800f7e60e60db"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.047829 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:13.060055 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:13.060766 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:13.060981 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:13.061614 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:13.062577 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:13.062654 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.062722 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:13.062772 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.071396 18168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41081
I20260620 09:09:13.071453 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41081 every 8 connection(s)
I20260620 09:09:13.072371 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:13.081190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18168
I20260620 09:09:13.081303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:09:13.081670 392 external_mini_cluster.cc:1428] Setting key ee3317ae2953730934b236ab8d4c0331
I20260620 09:09:13.082446 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.074388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42145 (local address 127.0.98.62:33313)
0620 09:09:13.074549 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.074554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.075826 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 09:09:13.077254 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.077257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.077308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.077396 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:13.078392 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.078969 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.080162 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.080381 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.080471 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.081883 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:13.081886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:13.081888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:13.081901 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:13.081922 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:13.081937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:13.082001 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.082203 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.082270 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:09:13.083464 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.074907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.2:42145)
0620 09:09:13.075642 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.075687 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.076993 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.077617 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.077625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.077717 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:13.078234 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.078247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.079114 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.079117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.080037 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.080048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.080564 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.080567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.080721 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.081560 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:13.081563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:13.081623 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:13.081717 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:13.082032 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:13.082040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:13.082046 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.082187 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.082506 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260620 09:09:13.084141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--builtin_ntp_servers=127.0.98.20:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:13.084764 18296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260620 09:09:13.085062 18296 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:13.085747 18296 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260620 09:09:13.086926 17976 ts_manager.cc:194] Registered new tserver with Master: a5bc576a944c4c24bbc60db19ec07d4a (127.0.98.2:41081)
I20260620 09:09:13.087793 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42145
W20260620 09:09:13.201431 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:13.201719 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:13.201761 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:13.206120 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:13.206185 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:13.206243 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:13.211103 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:13.212193 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:13.213658 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:13.223553 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:13.223572 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:13.223614 18299 server_base.cc:1034] running on GCE node
W20260620 09:09:13.223570 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:13.224179 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:13.224908 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:13.226102 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781946553226083 us; error 35 us; skew 500 ppm
I20260620 09:09:13.228057 18299 webserver.cc:460] Webserver started at http://127.0.98.3:41221/ using document root <none> and password file <none>
I20260620 09:09:13.228680 18299 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:13.228786 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:13.229028 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:13.231281 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "07935da713184a22b3b1e587f4a2c795"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "2d038695d2bb8dd33e9bdda353d57cf7"
server_key_iv: "b12a09a3a8a32c4305fbbd9eb5560d73"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.231868 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07935da713184a22b3b1e587f4a2c795"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "2d038695d2bb8dd33e9bdda353d57cf7"
server_key_iv: "b12a09a3a8a32c4305fbbd9eb5560d73"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.235867 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 09:09:13.238634 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.240017 18299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:13.240157 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "07935da713184a22b3b1e587f4a2c795"
format_stamp: "Formatted at 2026-06-20 09:09:13 on dist-test-slave-92t4"
server_key: "2d038695d2bb8dd33e9bdda353d57cf7"
server_key_iv: "b12a09a3a8a32c4305fbbd9eb5560d73"
server_key_version: "encryptionkey@0"
I20260620 09:09:13.240286 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:13.257236 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:13.257985 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:13.258198 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:13.258817 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:13.259817 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:13.259887 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.259963 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:13.260015 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:13.269109 18299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42553
I20260620 09:09:13.269129 18426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42553 every 8 connection(s)
I20260620 09:09:13.270059 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:13.271345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18299
I20260620 09:09:13.271464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:13.271737 392 external_mini_cluster.cc:1428] Setting key 0729acbff891a7f914b1f78979ff56dd
I20260620 09:09:13.280121 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.272028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34501 (local address 127.0.98.62:33313)
0620 09:09:13.272387 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.272392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.273585 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 09:09:13.275140 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.275143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.275199 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.275280 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:13.276235 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.276780 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.278055 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.278293 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.278411 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.279600 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:13.279604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:13.279607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:13.279620 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:13.279643 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:13.279657 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:13.279732 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.279893 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.279960 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260620 09:09:13.281131 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.272559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.3:34501)
0620 09:09:13.273442 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.273487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.274924 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.275464 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.275471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.275572 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:13.276090 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.276104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.276926 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.276930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.277917 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.277927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.278456 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.278460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.278642 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.279322 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:13.279323 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:13.279366 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:13.279434 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:13.279746 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:13.279759 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:13.279768 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.279895 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.280197 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260620 09:09:13.282356 18427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260620 09:09:13.282645 18427 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:13.283242 18427 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260620 09:09:13.284343 17976 ts_manager.cc:194] Registered new tserver with Master: 07935da713184a22b3b1e587f4a2c795 (127.0.98.3:42553)
I20260620 09:09:13.285190 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34501
I20260620 09:09:13.286118 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:13.291769 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.288110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.98.62:33313)
0620 09:09:13.288248 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.288252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.288379 (+ 127us) server_negotiation.cc:407] Connection header received
0620 09:09:13.288455 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.288457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.288497 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.288563 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:13.288800 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.289355 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.290149 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.290363 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.290456 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.291201 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:13.291204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:13.291206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:13.291215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:13.291231 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:13.291238 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:13.291302 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.291488 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.291609 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:09:13.294724 17976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:13.297806 17976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:13.312263 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.306290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.0.1:48456)
0620 09:09:13.306676 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.306732 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.307139 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.307951 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.307957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.307979 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:13.308147 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.308155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.309577 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.309580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.310773 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.310782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.311715 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.311718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.311825 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.311865 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.311971 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.312097 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":6,"thread_start_us":86,"threads_started":1}
I20260620 09:09:13.312318 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.306424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.1:45647)
0620 09:09:13.306943 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.306959 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.307292 (+ 333us) server_negotiation.cc:407] Connection header received
0620 09:09:13.307381 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.307388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.307618 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.307780 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:13.308274 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.309444 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.310903 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.311598 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.311705 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.311904 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.312017 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.312131 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":166,"threads_started":1}
I20260620 09:09:13.313405 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.307017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41081 (local address 127.0.0.1:45998)
0620 09:09:13.308410 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.308424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.308528 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.308944 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.308947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.308965 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:13.309074 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.309080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.310384 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.310387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.311439 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.311446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.312969 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.312978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.313099 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.313122 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.313194 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.313263 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":87,"threads_started":1}
I20260620 09:09:13.313421 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.307123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.98.3:42553)
0620 09:09:13.307717 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.307731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.307753 (+ 22us) server_negotiation.cc:407] Connection header received
0620 09:09:13.307825 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.307830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.308001 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.308151 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:13.308406 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.309386 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.311297 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.312122 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.312231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.312420 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.313155 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.313258 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":137,"threads_started":1}
I20260620 09:09:13.313736 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.306626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42553 (local address 127.0.0.1:58206)
0620 09:09:13.307591 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.307605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.307699 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.308143 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.308145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.308161 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:13.308276 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.308282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.309489 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.309495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.311160 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.311170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.312487 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.312491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.312833 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.312857 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.312930 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.312997 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":196,"thread_start_us":95,"threads_started":1}
I20260620 09:09:13.314002 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.307437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.98.2:41081)
0620 09:09:13.307833 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.307852 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.308535 (+ 683us) server_negotiation.cc:407] Connection header received
0620 09:09:13.308627 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.308633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.308803 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.308950 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:13.309169 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.310264 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.311559 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.312364 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.312732 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.312915 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.313707 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.313823 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":89,"threads_started":1}
I20260620 09:09:13.315721 18099 tablet_service.cc:1467] Processing CreateTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 (DEFAULT_TABLE table=test-table [id=9025410a73134ec28791838ada592ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:13.316895 18099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1137c7f3939d452fa6f65708b51a9c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:13.316576 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 (DEFAULT_TABLE table=test-table [id=9025410a73134ec28791838ada592ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:13.317698 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1137c7f3939d452fa6f65708b51a9c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:13.319144 18361 tablet_service.cc:1467] Processing CreateTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 (DEFAULT_TABLE table=test-table [id=9025410a73134ec28791838ada592ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:13.320151 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1137c7f3939d452fa6f65708b51a9c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:13.324405 18444 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Bootstrap starting.
I20260620 09:09:13.325675 18445 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Bootstrap starting.
I20260620 09:09:13.326735 18444 tablet_bootstrap.cc:654] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:13.327637 18444 log.cc:824] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:13.327672 18446 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Bootstrap starting.
I20260620 09:09:13.327935 18445 tablet_bootstrap.cc:654] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:13.328899 18445 log.cc:824] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:13.330199 18446 tablet_bootstrap.cc:654] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:13.330701 18445 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: No bootstrap required, opened a new log
I20260620 09:09:13.330749 18444 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: No bootstrap required, opened a new log
I20260620 09:09:13.330916 18445 ts_tablet_manager.cc:1397] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:09:13.330932 18444 ts_tablet_manager.cc:1397] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 09:09:13.331046 18446 log.cc:824] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:13.333112 18446 tablet_bootstrap.cc:492] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: No bootstrap required, opened a new log
I20260620 09:09:13.333303 18446 ts_tablet_manager.cc:1397] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:09:13.334731 18445 raft_consensus.cc:348] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.335013 18445 raft_consensus.cc:374] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:13.335071 18445 raft_consensus.cc:729] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bc576a944c4c24bbc60db19ec07d4a, State: Initialized, Role: FOLLOWER
I20260620 09:09:13.335322 18444 raft_consensus.cc:348] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.335590 18444 raft_consensus.cc:374] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:13.335541 18445 consensus_queue.cc:260] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.335650 18444 raft_consensus.cc:729] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db824b506634a2e994ba54502cbcea8, State: Initialized, Role: FOLLOWER
I20260620 09:09:13.336087 18444 consensus_queue.cc:260] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.336644 18445 ts_tablet_manager.cc:1428] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:13.336799 18165 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
I20260620 09:09:13.336936 18296 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
I20260620 09:09:13.337117 18444 ts_tablet_manager.cc:1428] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 09:09:13.337492 18446 raft_consensus.cc:348] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.337760 18446 raft_consensus.cc:374] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:13.337846 18446 raft_consensus.cc:729] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07935da713184a22b3b1e587f4a2c795, State: Initialized, Role: FOLLOWER
I20260620 09:09:13.338240 18446 consensus_queue.cc:260] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.338889 18427 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
I20260620 09:09:13.339228 18446 ts_tablet_manager.cc:1428] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 09:09:13.382450 18166 tablet.cc:2369] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:13.394078 18450 raft_consensus.cc:482] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:13.394342 18450 raft_consensus.cc:504] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.395608 18450 leader_election.cc:288] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db824b506634a2e994ba54502cbcea8 (127.0.98.1:45647), 07935da713184a22b3b1e587f4a2c795 (127.0.98.3:42553)
I20260620 09:09:13.401146 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.396223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40489 (local address 127.0.98.1:45647)
0620 09:09:13.396348 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.396354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.396371 (+ 17us) server_negotiation.cc:407] Connection header received
0620 09:09:13.396434 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.396436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.396477 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.396615 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:13.396939 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.397792 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.399683 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.400284 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.400399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.400465 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.400889 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.400970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:09:13.401146 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.396194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40685 (local address 127.0.98.3:42553)
0620 09:09:13.396325 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.396330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.396538 (+ 208us) server_negotiation.cc:407] Connection header received
0620 09:09:13.396607 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.396610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.396655 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.396731 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:13.397065 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.397792 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.399683 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.400284 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.400399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.400465 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.400889 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.400970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:09:13.401233 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.395915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.98.2:40489)
0620 09:09:13.396217 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.396233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.396339 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.396643 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.396650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.396663 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:13.396809 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.396816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.397954 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.397959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.399542 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.399556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.400463 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.400466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.400691 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.400766 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.400863 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.401031 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":328}
I20260620 09:09:13.401242 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.395915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42553 (local address 127.0.98.2:40685)
0620 09:09:13.396431 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.396453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:13.396527 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:13.396755 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:13.396757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:13.396777 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:13.396892 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.396901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.397954 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.397959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.399550 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:13.399556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.400459 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.400464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:13.400718 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.400769 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 09:09:13.400863 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 09:09:13.401031 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":319,"thread_start_us":126,"threads_started":1}
I20260620 09:09:13.401988 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1137c7f3939d452fa6f65708b51a9c17" candidate_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07935da713184a22b3b1e587f4a2c795" is_pre_election: true
I20260620 09:09:13.401981 18119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1137c7f3939d452fa6f65708b51a9c17" candidate_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db824b506634a2e994ba54502cbcea8" is_pre_election: true
I20260620 09:09:13.402315 18381 raft_consensus.cc:2454] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bc576a944c4c24bbc60db19ec07d4a in term 0.
I20260620 09:09:13.402336 18119 raft_consensus.cc:2454] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bc576a944c4c24bbc60db19ec07d4a in term 0.
I20260620 09:09:13.402891 18186 leader_election.cc:302] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07935da713184a22b3b1e587f4a2c795, 0db824b506634a2e994ba54502cbcea8, a5bc576a944c4c24bbc60db19ec07d4a; no voters:
I20260620 09:09:13.403246 18450 raft_consensus.cc:2781] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:13.403348 18450 raft_consensus.cc:482] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:13.403429 18450 raft_consensus.cc:3037] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:13.404532 18450 raft_consensus.cc:504] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.404930 18450 leader_election.cc:288] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [CANDIDATE]: Term 1 election: Requested vote from peers 0db824b506634a2e994ba54502cbcea8 (127.0.98.1:45647), 07935da713184a22b3b1e587f4a2c795 (127.0.98.3:42553)
I20260620 09:09:13.405460 18119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1137c7f3939d452fa6f65708b51a9c17" candidate_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db824b506634a2e994ba54502cbcea8"
I20260620 09:09:13.405588 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1137c7f3939d452fa6f65708b51a9c17" candidate_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07935da713184a22b3b1e587f4a2c795"
I20260620 09:09:13.405597 18119 raft_consensus.cc:3037] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:13.405750 18381 raft_consensus.cc:3037] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:13.406777 18119 raft_consensus.cc:2454] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bc576a944c4c24bbc60db19ec07d4a in term 1.
I20260620 09:09:13.406884 18381 raft_consensus.cc:2454] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bc576a944c4c24bbc60db19ec07d4a in term 1.
I20260620 09:09:13.407198 18185 leader_election.cc:302] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db824b506634a2e994ba54502cbcea8, a5bc576a944c4c24bbc60db19ec07d4a; no voters:
I20260620 09:09:13.407385 18450 raft_consensus.cc:2781] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:13.407649 18450 raft_consensus.cc:686] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 1 LEADER]: Becoming Leader. State: Replica: a5bc576a944c4c24bbc60db19ec07d4a, State: Running, Role: LEADER
I20260620 09:09:13.408051 18450 consensus_queue.cc:237] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } }
I20260620 09:09:13.411707 17976 catalog_manager.cc:5495] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to a5bc576a944c4c24bbc60db19ec07d4a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5bc576a944c4c24bbc60db19ec07d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 } health_report { overall_health: UNKNOWN } } }
I20260620 09:09:13.442953 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.439208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.2:41081)
0620 09:09:13.439350 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:13.439355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:13.439371 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:09:13.439499 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:13.439501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:13.439560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:13.439681 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:13.439946 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.440476 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.441437 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:13.441660 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:13.441770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:13.442100 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:13.442539 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:13.442680 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:13.442717 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 09:09:13.442774 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 09:09:13.449390 18381 raft_consensus.cc:1261] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 1 FOLLOWER]: Refusing update from remote peer a5bc576a944c4c24bbc60db19ec07d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:13.449391 18119 raft_consensus.cc:1261] T 1137c7f3939d452fa6f65708b51a9c17 P 0db824b506634a2e994ba54502cbcea8 [term 1 FOLLOWER]: Refusing update from remote peer a5bc576a944c4c24bbc60db19ec07d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:13.450197 18454 consensus_queue.cc:1035] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db824b506634a2e994ba54502cbcea8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:13.450397 18450 consensus_queue.cc:1035] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07935da713184a22b3b1e587f4a2c795" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:13.457664 18457 mvcc.cc:205] Tried to move back new op lower bound from 7298853082920394752 to 7298853082762584064. Current Snapshot: MvccSnapshot[applied={T|T < 7298853082920394752}]
I20260620 09:09:13.458973 18458 mvcc.cc:205] Tried to move back new op lower bound from 7298853082920394752 to 7298853082762584064. Current Snapshot: MvccSnapshot[applied={T|T < 7298853082920394752}]
I20260620 09:09:13.462591 18459 mvcc.cc:205] Tried to move back new op lower bound from 7298853082920394752 to 7298853082762584064. Current Snapshot: MvccSnapshot[applied={T|T < 7298853082920394752 or (T in {7298853082920394752})}]
I20260620 09:09:13.467005 17975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:13.467206 17975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:13.469894 17975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89e5c1c418e94353a62915539489037e: Sending DeleteTablet for 3 replicas of tablet 1137c7f3939d452fa6f65708b51a9c17
I20260620 09:09:13.470866 18361 tablet_service.cc:1514] Processing DeleteTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:13 UTC) from {username='slave'} at 127.0.0.1:58206
I20260620 09:09:13.470883 18099 tablet_service.cc:1514] Processing DeleteTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:13 UTC) from {username='slave'} at 127.0.0.1:48456
I20260620 09:09:13.471151 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 1137c7f3939d452fa6f65708b51a9c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:13 UTC) from {username='slave'} at 127.0.0.1:45998
I20260620 09:09:13.471607 18468 tablet_replica.cc:330] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: stopping tablet replica
I20260620 09:09:13.471622 18466 tablet_replica.cc:330] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: stopping tablet replica
I20260620 09:09:13.471961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18037
I20260620 09:09:13.471987 18466 raft_consensus.cc:2229] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:13.471985 18468 raft_consensus.cc:2229] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:13.473585 18466 raft_consensus.cc:2258] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:13.474028 18468 raft_consensus.cc:2258] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:13.475669 18466 ts_tablet_manager.cc:1905] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:13.476480 18468 ts_tablet_manager.cc:1905] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:13.478574 18466 ts_tablet_manager.cc:1918] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:13.478679 18466 log.cc:1196] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/1137c7f3939d452fa6f65708b51a9c17
I20260620 09:09:13.478986 18466 ts_tablet_manager.cc:1939] T 1137c7f3939d452fa6f65708b51a9c17 P 07935da713184a22b3b1e587f4a2c795: Deleting consensus metadata
I20260620 09:09:13.480268 18468 ts_tablet_manager.cc:1918] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:13.480434 18468 log.cc:1196] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/1137c7f3939d452fa6f65708b51a9c17
I20260620 09:09:13.480403 17964 catalog_manager.cc:4841] TS 07935da713184a22b3b1e587f4a2c795 (127.0.98.3:42553): tablet 1137c7f3939d452fa6f65708b51a9c17 (table test-table [id=9025410a73134ec28791838ada592ddb]) successfully deleted
W20260620 09:09:13.480533 17963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45647 (error 108)
I20260620 09:09:13.480842 18468 ts_tablet_manager.cc:1939] T 1137c7f3939d452fa6f65708b51a9c17 P a5bc576a944c4c24bbc60db19ec07d4a: Deleting consensus metadata
I20260620 09:09:13.481874 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.481413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45647 (local address 127.0.0.1:48460)
0620 09:09:13.481707 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.481781 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 09:09:13.481882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18168
W20260620 09:09:13.483522 17963 catalog_manager.cc:4584] TS 0db824b506634a2e994ba54502cbcea8 (127.0.98.1:45647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1137c7f3939d452fa6f65708b51a9c17: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45647: connect: Connection refused (error 111)
I20260620 09:09:13.487958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18299
I20260620 09:09:13.488903 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:13.488184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41081 (local address 127.0.0.1:46016)
0620 09:09:13.488720 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:13.488788 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
I20260620 09:09:13.493768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 17946
2026-06-20T09:09:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1078 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:13 dist-test-slave-92t4 krb5kdc[18472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:13 dist-test-slave-92t4 krb5kdc[18472](info): set up 2 sockets
Jun 20 09:09:13 dist-test-slave-92t4 krb5kdc[18472](info): commencing operation
krb5kdc: starting...
W20260620 09:09:14.536957 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:14 dist-test-slave-92t4 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:14.664220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:14.778452 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:14.778759 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:14.778800 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:14.782958 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:14.783025 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:14.783046 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:14.783062 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:14.783119 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:14.788769 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:14.789966 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:14.791028 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:14.800048 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:14.800014 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:14.800084 18485 server_base.cc:1034] running on GCE node
W20260620 09:09:14.800031 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:14.800850 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:14.802127 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:14.803334 18485 hybrid_clock.cc:648] HybridClock initialized: now 1781946554803297 us; error 49 us; skew 500 ppm
Jun 20 09:09:14 dist-test-slave-92t4 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:14.810194 18485 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:14.811235 18485 webserver.cc:460] Webserver started at http://127.0.98.62:33331/ using document root <none> and password file <none>
I20260620 09:09:14.811776 18485 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:14.811836 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:14.812012 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:14.814165 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "e378ecc7e7a64042a1c6dcdccc6fc336"
format_stamp: "Formatted at 2026-06-20 09:09:14 on dist-test-slave-92t4"
server_key: "245f0fb0b82194bb5cc5f5b9d83bfd67"
server_key_iv: "be027003c291d7f78ea35a7228986c55"
server_key_version: "encryptionkey@0"
I20260620 09:09:14.814698 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "e378ecc7e7a64042a1c6dcdccc6fc336"
format_stamp: "Formatted at 2026-06-20 09:09:14 on dist-test-slave-92t4"
server_key: "245f0fb0b82194bb5cc5f5b9d83bfd67"
server_key_iv: "be027003c291d7f78ea35a7228986c55"
server_key_version: "encryptionkey@0"
I20260620 09:09:14.818603 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:09:14.821180 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:14.822515 18485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:14.822659 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "e378ecc7e7a64042a1c6dcdccc6fc336"
format_stamp: "Formatted at 2026-06-20 09:09:14 on dist-test-slave-92t4"
server_key: "245f0fb0b82194bb5cc5f5b9d83bfd67"
server_key_iv: "be027003c291d7f78ea35a7228986c55"
server_key_version: "encryptionkey@0"
I20260620 09:09:14.822781 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:14.848070 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:14.852587 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:14.852828 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:14.863001 18485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33593
I20260620 09:09:14.863016 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33593 every 8 connection(s)
I20260620 09:09:14.864516 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:14.867564 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:14.871497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18485
I20260620 09:09:14.871658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:14.871975 392 external_mini_cluster.cc:1428] Setting key 0e75259a920bbe9176efdf93f211d74d
I20260620 09:09:14.874476 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: Bootstrap starting.
I20260620 09:09:14.877244 18553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:14.878109 18553 log.cc:824] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:14.880178 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: No bootstrap required, opened a new log
Jun 20 09:09:14 dist-test-slave-92t4 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:14.884442 18553 raft_consensus.cc:348] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260620 09:09:14.884717 18553 raft_consensus.cc:374] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:14.884800 18553 raft_consensus.cc:729] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e378ecc7e7a64042a1c6dcdccc6fc336, State: Initialized, Role: FOLLOWER
I20260620 09:09:14.885390 18553 consensus_queue.cc:260] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260620 09:09:14.885558 18553 raft_consensus.cc:388] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:14.885645 18553 raft_consensus.cc:482] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:14.885753 18553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:14.886731 18553 raft_consensus.cc:504] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260620 09:09:14.887107 18553 leader_election.cc:302] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e378ecc7e7a64042a1c6dcdccc6fc336; no voters:
I20260620 09:09:14.887482 18553 leader_election.cc:288] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:14.887563 18558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:14.887768 18558 raft_consensus.cc:686] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [term 1 LEADER]: Becoming Leader. State: Replica: e378ecc7e7a64042a1c6dcdccc6fc336, State: Running, Role: LEADER
I20260620 09:09:14.888099 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:14.873543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.0.98.62:33593)
0620 09:09:14.873979 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:14.874004 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:14.874057 (+ 53us) server_negotiation.cc:407] Connection header received
0620 09:09:14.875192 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:14.875223 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:14.875773 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:14.876263 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:14.877238 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:14.878604 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:14.879512 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:14.880012 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:14.880198 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:14.882768 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:14.882773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:14.882788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:14.882815 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:14.884577 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:14.885220 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:14.885226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:14.885232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:14.885406 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:14.885746 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:14.885749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:14.885751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:14.886232 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:14.886473 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:14.886761 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 09:09:14.887100 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260620 09:09:14.888168 18558 consensus_queue.cc:237] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260620 09:09:14.888821 18553 sys_catalog.cc:564] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:14.890265 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } } }
I20260620 09:09:14.890691 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:14.890596 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e378ecc7e7a64042a1c6dcdccc6fc336. Latest consensus state: current_term: 1 leader_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e378ecc7e7a64042a1c6dcdccc6fc336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } } }
I20260620 09:09:14.891404 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336 [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:14.894065 18572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:14.894182 18572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:14.894295 18573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:14.895148 18573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:14.901530 18573 catalog_manager.cc:1347] Generated new cluster ID: be4eb61daa104c129cbd07b8905e5759
I20260620 09:09:14.901603 18573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:14.915571 18573 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:14.916646 18573 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:14.924440 18573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: Generated new TSK 0
I20260620 09:09:14.925359 18573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:09:14.963701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:15.107808 18579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:15.108105 18579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:15.108152 18579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:15.112439 18579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:15.112517 18579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:15.112582 18579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:15.118062 18579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:15.119238 18579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:15.120262 18579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:15.130396 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.130402 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.130405 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:15.130491 18579 server_base.cc:1034] running on GCE node
I20260620 09:09:15.131131 18579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:15.131858 18579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:15.133025 18579 hybrid_clock.cc:648] HybridClock initialized: now 1781946555132988 us; error 53 us; skew 500 ppm
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:15.141618 18579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:09:15.142612 18579 webserver.cc:460] Webserver started at http://127.0.98.1:35137/ using document root <none> and password file <none>
I20260620 09:09:15.143201 18579 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:15.143265 18579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:15.143443 18579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:15.145713 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "74b89cc6283342f6a509c8102abf3317"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "337d798c8a2791f259c5bd7d23ead041"
server_key_iv: "84a2a490e7942d69ea40112bd8fd550f"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.146239 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74b89cc6283342f6a509c8102abf3317"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "337d798c8a2791f259c5bd7d23ead041"
server_key_iv: "84a2a490e7942d69ea40112bd8fd550f"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.150125 18579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:09:15.152647 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.154119 18579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:15.154276 18579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "74b89cc6283342f6a509c8102abf3317"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "337d798c8a2791f259c5bd7d23ead041"
server_key_iv: "84a2a490e7942d69ea40112bd8fd550f"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.154402 18579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:15.169153 18579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:15.172364 18579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:15.172590 18579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:15.173295 18579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:15.174206 18579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:15.174279 18579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.174355 18579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:15.174408 18579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.183763 18579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34611
I20260620 09:09:15.183785 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34611 every 8 connection(s)
I20260620 09:09:15.184716 18579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:15.191802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18579
I20260620 09:09:15.191920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:15.192242 392 external_mini_cluster.cc:1428] Setting key 195753a6a00dbbd873ef975709c0fa6b
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:15.200117 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.186825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50119 (local address 127.0.98.62:33593)
0620 09:09:15.186970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.186982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.187823 (+ 841us) server_negotiation.cc:407] Connection header received
0620 09:09:15.189283 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.189286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.189344 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.189430 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:15.190676 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.191246 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.192623 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.192875 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.192996 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.195726 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:15.195730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:15.195732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:15.195760 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:15.197541 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.198254 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.198257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.198258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.198343 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.198795 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.198797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.198798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.198997 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:15.199077 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.199860 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.199980 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 09:09:15.201536 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.187223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.1:50119)
0620 09:09:15.187670 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.187715 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.188958 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.189618 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.189626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.190009 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:15.190514 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.190527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.191384 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.191387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.192476 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.192486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.193089 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.193095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.193260 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.193947 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:15.193964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:15.195520 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:15.197682 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.197691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.197712 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.198142 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.198448 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.198454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.198459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.198676 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.199115 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:15.199126 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:15.199565 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.199830 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.200253 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:09:15.202932 18708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260620 09:09:15.203235 18708 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:15.203769 18708 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 09:09:15.205495 18517 ts_manager.cc:194] Registered new tserver with Master: 74b89cc6283342f6a509c8102abf3317 (127.0.98.1:34611)
I20260620 09:09:15.206667 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:09:15.220685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:15.337102 18713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:15.337416 18713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:15.337461 18713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:15.341631 18713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:15.341703 18713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:15.341766 18713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:15.347257 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:15.348380 18713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:15.349479 18713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:15.360002 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.360009 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.360018 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:15.360435 18713 server_base.cc:1034] running on GCE node
I20260620 09:09:15.360862 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:15.361614 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:15.362780 18713 hybrid_clock.cc:648] HybridClock initialized: now 1781946555362736 us; error 58 us; skew 500 ppm
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:15.365670 18713 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:09:15.366562 18713 webserver.cc:460] Webserver started at http://127.0.98.2:36275/ using document root <none> and password file <none>
I20260620 09:09:15.367154 18713 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:15.367215 18713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:15.367384 18713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:15.369489 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "db898662bb384c3c858499d0016d37ad"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "c8f2ef1e2b4d58564bfbf4ea3c127ca1"
server_key_iv: "f03e528838dc367b75ef408f6b7ab396"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.370036 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db898662bb384c3c858499d0016d37ad"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "c8f2ef1e2b4d58564bfbf4ea3c127ca1"
server_key_iv: "f03e528838dc367b75ef408f6b7ab396"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.373821 18713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 09:09:15.376350 18728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.377774 18713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:09:15.377874 18713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "db898662bb384c3c858499d0016d37ad"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "c8f2ef1e2b4d58564bfbf4ea3c127ca1"
server_key_iv: "f03e528838dc367b75ef408f6b7ab396"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.378021 18713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:15.411412 18713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:15.414997 18713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:15.415211 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:15.415902 18713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:15.416782 18713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:15.416828 18713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.416868 18713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:15.416884 18713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.426653 18713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34749
I20260620 09:09:15.426672 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34749 every 8 connection(s)
I20260620 09:09:15.427801 18713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:15.438014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18713
I20260620 09:09:15.438163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:15.438486 392 external_mini_cluster.cc:1428] Setting key e2d8c5340167727c61d1dec01638568b
I20260620 09:09:15.443222 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.429891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55693 (local address 127.0.98.62:33593)
0620 09:09:15.430051 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.430057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.431169 (+ 1112us) server_negotiation.cc:407] Connection header received
0620 09:09:15.432495 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.432498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.432547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.432632 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:15.433999 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.434546 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.435937 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.436173 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.436264 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.439173 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:15.439176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:15.439178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:15.439204 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:15.440815 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.441445 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.441449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.441450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.441532 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.441885 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.441888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.441890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.442082 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:15.442184 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.442928 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.443061 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:09:15.444208 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.430482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.2:55693)
0620 09:09:15.430996 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.431044 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.432277 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.432806 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.432814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.433335 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:15.433852 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.433866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.434664 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.434667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.435771 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.435786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.436326 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.436329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.436501 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.437207 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:15.437225 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:15.438956 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:15.440966 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.440972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.440988 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.441323 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.441632 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.441635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.441637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.441784 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.442199 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:15.442207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:15.442644 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.442915 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.443301 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260620 09:09:15.445678 18842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260620 09:09:15.445956 18842 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:15.446507 18842 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
I20260620 09:09:15.447670 18517 ts_manager.cc:194] Registered new tserver with Master: db898662bb384c3c858499d0016d37ad (127.0.98.2:34749)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:09:15.448328 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:09:15.464952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:15.579420 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:15.579701 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:15.579746 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:15.584060 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:15.584127 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:15.584223 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:15.589948 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:15.591176 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:15.592234 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:15.602029 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.602109 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:15.602018 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:15.602211 18847 server_base.cc:1034] running on GCE node
I20260620 09:09:15.602713 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:15.603317 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:15.604498 18847 hybrid_clock.cc:648] HybridClock initialized: now 1781946555604478 us; error 30 us; skew 500 ppm
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:15.607601 18847 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:09:15.608501 18847 webserver.cc:460] Webserver started at http://127.0.98.3:33929/ using document root <none> and password file <none>
I20260620 09:09:15.609133 18847 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:15.609226 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:15.609467 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:15.611578 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "cead037095cf4ec29fcff2deaf557bc7"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "bd60f00e20fd2ba35c20ee55364bc3fc"
server_key_iv: "1d05cededbbcdecbacfff5eb5347d833"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.612190 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cead037095cf4ec29fcff2deaf557bc7"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "bd60f00e20fd2ba35c20ee55364bc3fc"
server_key_iv: "1d05cededbbcdecbacfff5eb5347d833"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.616113 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:09:15.618659 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.619936 18847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:15.620079 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "cead037095cf4ec29fcff2deaf557bc7"
format_stamp: "Formatted at 2026-06-20 09:09:15 on dist-test-slave-92t4"
server_key: "bd60f00e20fd2ba35c20ee55364bc3fc"
server_key_iv: "1d05cededbbcdecbacfff5eb5347d833"
server_key_version: "encryptionkey@0"
I20260620 09:09:15.620193 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:15.640661 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:15.644246 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:15.644467 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:15.645171 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:15.646049 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:15.646121 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.646194 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:15.646236 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:15.655156 18847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35205
I20260620 09:09:15.655180 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35205 every 8 connection(s)
I20260620 09:09:15.656041 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:15.661619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 18847
I20260620 09:09:15.661762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:15.662038 392 external_mini_cluster.cc:1428] Setting key 974ada240ad70189760ac47f1c61e9d6
Jun 20 09:09:15 dist-test-slave-92t4 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:15.670202 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.657987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43663 (local address 127.0.98.62:33593)
0620 09:09:15.658131 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.658135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.659056 (+ 921us) server_negotiation.cc:407] Connection header received
0620 09:09:15.660303 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.660307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.660361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.660485 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:15.661848 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.662471 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.663953 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.664233 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.664353 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.666926 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:15.666930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:15.666932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:15.666955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:15.668318 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.668780 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.668782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.668783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.668859 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.669216 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.669219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.669220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.669401 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:15.669481 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.669946 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.670078 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 09:09:15.671012 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.658403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.3:43663)
0620 09:09:15.658887 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.658952 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.660102 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.660656 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.660663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.661015 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:15.661676 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.661697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.662574 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.662577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.663771 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.663780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.664414 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.664417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.664576 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.665277 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:15.665294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:15.666663 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:15.668424 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.668429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.668442 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.668689 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.668938 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:15.668940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:15.668941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:15.669136 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:15.669493 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:15.669500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:15.669734 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.669888 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.670164 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260620 09:09:15.672240 18976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260620 09:09:15.672487 18976 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:15.672976 18976 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
I20260620 09:09:15.674031 18517 ts_manager.cc:194] Registered new tserver with Master: cead037095cf4ec29fcff2deaf557bc7 (127.0.98.3:35205)
I20260620 09:09:15.674737 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43663
I20260620 09:09:15.675796 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:15.685962 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.677909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.0.98.62:33593)
0620 09:09:15.678068 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.678071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.678148 (+ 77us) server_negotiation.cc:407] Connection header received
0620 09:09:15.678241 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.678244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.678295 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.678361 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:15.679230 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.679738 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.680583 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.680791 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.680898 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.682956 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:15.682959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:15.682962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:15.682984 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:15.684183 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.684643 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.684645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.684647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.684717 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:15.685112 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:15.685115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:15.685117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:15.685318 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:15.685436 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.685680 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.685781 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 09:09:15.689122 18517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:15.692366 18517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:15.707314 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.700867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.98.3:35205)
0620 09:09:15.701291 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.701303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.701736 (+ 433us) server_negotiation.cc:407] Connection header received
0620 09:09:15.702019 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.702026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.702238 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.702375 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:15.702994 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.703825 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.705694 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.706457 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.706573 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.706717 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.707049 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.707148 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260620 09:09:15.707309 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.700748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35205 (local address 127.0.0.1:49600)
0620 09:09:15.701599 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.701649 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.701793 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.702599 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.702605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.702627 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:15.702788 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.702793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.703951 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.703955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.705531 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.705539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.706584 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.706587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.706780 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.706877 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.706998 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.707160 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":109,"thread_start_us":74,"threads_started":1}
I20260620 09:09:15.707815 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.700839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34749 (local address 127.0.0.1:53914)
0620 09:09:15.701795 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.701808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.701914 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.702656 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.702659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.702676 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:15.702783 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.702790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.704399 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.704402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.705501 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.705511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.706584 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.706587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.706746 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.706878 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.707021 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.707674 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":613,"thread_start_us":125,"threads_started":1}
I20260620 09:09:15.708120 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.702212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.0.98.1:34611)
0620 09:09:15.702493 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.702499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.702519 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:15.702592 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.702597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.702762 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.702912 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:15.703122 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.704237 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.706781 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.707572 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.707698 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.707833 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.707891 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.707987 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 09:09:15.708344 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.701481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34611 (local address 127.0.0.1:43308)
0620 09:09:15.701970 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.701983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.702057 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.702896 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.702901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.702923 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:15.702998 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.703003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.704363 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.704367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.705595 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.705605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.707698 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.707701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.707784 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.707804 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.707869 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.707938 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":78,"thread_start_us":90,"threads_started":1}
I20260620 09:09:15.708781 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.701305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.98.2:34749)
0620 09:09:15.701810 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.701823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.702037 (+ 214us) server_negotiation.cc:407] Connection header received
0620 09:09:15.702120 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.702128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.702339 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.702497 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:15.703051 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.704237 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.705603 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.706460 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.706573 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.706717 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.707141 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.707238 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":111,"threads_started":1}
I20260620 09:09:15.711038 18642 tablet_service.cc:1467] Processing CreateTablet for tablet e1e870b40ef44f6793b6c11285043abd (DEFAULT_TABLE table=test-table [id=54d65afb247a408e966c2828de7a6224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:15.710953 18910 tablet_service.cc:1467] Processing CreateTablet for tablet e1e870b40ef44f6793b6c11285043abd (DEFAULT_TABLE table=test-table [id=54d65afb247a408e966c2828de7a6224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:15.711978 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e870b40ef44f6793b6c11285043abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:15.712013 18910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e870b40ef44f6793b6c11285043abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:15.711643 18776 tablet_service.cc:1467] Processing CreateTablet for tablet e1e870b40ef44f6793b6c11285043abd (DEFAULT_TABLE table=test-table [id=54d65afb247a408e966c2828de7a6224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:15.712555 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e870b40ef44f6793b6c11285043abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:15.718662 18993 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: Bootstrap starting.
I20260620 09:09:15.720165 18995 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Bootstrap starting.
I20260620 09:09:15.721304 18993 tablet_bootstrap.cc:654] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:15.721410 18994 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Bootstrap starting.
I20260620 09:09:15.722074 18993 log.cc:824] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:15.722317 18995 tablet_bootstrap.cc:654] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:15.723548 18994 tablet_bootstrap.cc:654] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:15.723560 18995 log.cc:824] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:15.724233 18993 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: No bootstrap required, opened a new log
I20260620 09:09:15.724454 18993 ts_tablet_manager.cc:1397] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:09:15.724697 18994 log.cc:824] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:15.725896 18995 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: No bootstrap required, opened a new log
I20260620 09:09:15.726092 18995 ts_tablet_manager.cc:1397] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:09:15.726600 18994 tablet_bootstrap.cc:492] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: No bootstrap required, opened a new log
I20260620 09:09:15.726787 18994 ts_tablet_manager.cc:1397] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 09:09:15.729195 18993 raft_consensus.cc:348] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.729501 18993 raft_consensus.cc:374] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:15.729558 18993 raft_consensus.cc:729] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b89cc6283342f6a509c8102abf3317, State: Initialized, Role: FOLLOWER
I20260620 09:09:15.729928 18994 raft_consensus.cc:348] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.730028 18993 consensus_queue.cc:260] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.730166 18994 raft_consensus.cc:374] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:15.730224 18994 raft_consensus.cc:729] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db898662bb384c3c858499d0016d37ad, State: Initialized, Role: FOLLOWER
I20260620 09:09:15.730720 18708 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
I20260620 09:09:15.730682 18994 consensus_queue.cc:260] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.731132 18993 ts_tablet_manager.cc:1428] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 09:09:15.730010 18995 raft_consensus.cc:348] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.731462 18995 raft_consensus.cc:374] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:15.731516 18995 raft_consensus.cc:729] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cead037095cf4ec29fcff2deaf557bc7, State: Initialized, Role: FOLLOWER
I20260620 09:09:15.731643 18994 ts_tablet_manager.cc:1428] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260620 09:09:15.731742 18842 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
I20260620 09:09:15.731943 18995 consensus_queue.cc:260] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.732693 18976 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
I20260620 09:09:15.732961 18995 ts_tablet_manager.cc:1428] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 09:09:15.848634 19001 raft_consensus.cc:482] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:15.848891 19001 raft_consensus.cc:504] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.850207 19001 leader_election.cc:288] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db898662bb384c3c858499d0016d37ad (127.0.98.2:34749), 74b89cc6283342f6a509c8102abf3317 (127.0.98.1:34611)
I20260620 09:09:15.854730 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.850646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32867 (local address 127.0.98.2:34749)
0620 09:09:15.850783 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.850788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.851005 (+ 217us) server_negotiation.cc:407] Connection header received
0620 09:09:15.851149 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.851152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.851202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.851294 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:15.851564 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.852119 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.853531 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.854172 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.854265 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.854318 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.854533 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.854600 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:09:15.854730 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.850607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44743 (local address 127.0.98.1:34611)
0620 09:09:15.850719 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.850725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.850838 (+ 113us) server_negotiation.cc:407] Connection header received
0620 09:09:15.850918 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.850920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.850966 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.851082 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:15.851360 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.851875 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.853439 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.854095 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.854224 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.854298 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.854525 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.854602 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:09:15.854785 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.850511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34611 (local address 127.0.98.3:44743)
0620 09:09:15.850710 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.850726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.850834 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.851091 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.851094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.851112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:15.851234 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.851240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.851996 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.852001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.853295 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.853305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.854214 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.854238 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.854352 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.854407 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.854487 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.854624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":42}
I20260620 09:09:15.854977 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.850515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34749 (local address 127.0.98.3:32867)
0620 09:09:15.850891 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.850904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:15.851034 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:15.851313 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:15.851316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:15.851331 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:15.851454 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.851459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.852237 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.852240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.853391 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:15.853403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.854276 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.854280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:15.854367 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.854422 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 09:09:15.854487 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 09:09:15.854624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":94,"thread_start_us":134,"threads_started":1}
I20260620 09:09:15.855479 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e870b40ef44f6793b6c11285043abd" candidate_uuid: "cead037095cf4ec29fcff2deaf557bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b89cc6283342f6a509c8102abf3317" is_pre_election: true
I20260620 09:09:15.855506 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e870b40ef44f6793b6c11285043abd" candidate_uuid: "cead037095cf4ec29fcff2deaf557bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db898662bb384c3c858499d0016d37ad" is_pre_election: true
I20260620 09:09:15.855791 18796 raft_consensus.cc:2454] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cead037095cf4ec29fcff2deaf557bc7 in term 0.
I20260620 09:09:15.855794 18662 raft_consensus.cc:2454] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cead037095cf4ec29fcff2deaf557bc7 in term 0.
I20260620 09:09:15.856298 18866 leader_election.cc:302] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cead037095cf4ec29fcff2deaf557bc7, db898662bb384c3c858499d0016d37ad; no voters:
I20260620 09:09:15.856606 19001 raft_consensus.cc:2781] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:15.856698 19001 raft_consensus.cc:482] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:15.856743 19001 raft_consensus.cc:3037] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:15.857777 19001 raft_consensus.cc:504] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.858109 19001 leader_election.cc:288] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [CANDIDATE]: Term 1 election: Requested vote from peers db898662bb384c3c858499d0016d37ad (127.0.98.2:34749), 74b89cc6283342f6a509c8102abf3317 (127.0.98.1:34611)
I20260620 09:09:15.858430 18796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e870b40ef44f6793b6c11285043abd" candidate_uuid: "cead037095cf4ec29fcff2deaf557bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db898662bb384c3c858499d0016d37ad"
I20260620 09:09:15.858543 18796 raft_consensus.cc:3037] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:15.859568 18796 raft_consensus.cc:2454] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cead037095cf4ec29fcff2deaf557bc7 in term 1.
I20260620 09:09:15.859802 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e870b40ef44f6793b6c11285043abd" candidate_uuid: "cead037095cf4ec29fcff2deaf557bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b89cc6283342f6a509c8102abf3317"
I20260620 09:09:15.859823 18866 leader_election.cc:302] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cead037095cf4ec29fcff2deaf557bc7, db898662bb384c3c858499d0016d37ad; no voters:
I20260620 09:09:15.859903 18662 raft_consensus.cc:3037] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:15.860009 19001 raft_consensus.cc:2781] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:15.860251 19001 raft_consensus.cc:686] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 1 LEADER]: Becoming Leader. State: Replica: cead037095cf4ec29fcff2deaf557bc7, State: Running, Role: LEADER
I20260620 09:09:15.860584 19001 consensus_queue.cc:237] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } }
I20260620 09:09:15.861138 18662 raft_consensus.cc:2454] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cead037095cf4ec29fcff2deaf557bc7 in term 1.
I20260620 09:09:15.864346 18517 catalog_manager.cc:5495] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cead037095cf4ec29fcff2deaf557bc7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cead037095cf4ec29fcff2deaf557bc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cead037095cf4ec29fcff2deaf557bc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } health_report { overall_health: UNKNOWN } } }
I20260620 09:09:15.898969 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.895172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.0.98.3:35205)
0620 09:09:15.895321 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:15.895325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:15.895341 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:09:15.895481 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:15.895484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:15.895539 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:15.895661 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:15.895956 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.896456 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.897420 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:15.897639 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:15.897753 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:15.898099 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:15.898510 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:15.898674 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:15.898757 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:09:15.898802 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:09:15.905155 18796 raft_consensus.cc:1261] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 1 FOLLOWER]: Refusing update from remote peer cead037095cf4ec29fcff2deaf557bc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:15.905155 18662 raft_consensus.cc:1261] T e1e870b40ef44f6793b6c11285043abd P 74b89cc6283342f6a509c8102abf3317 [term 1 FOLLOWER]: Refusing update from remote peer cead037095cf4ec29fcff2deaf557bc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:15.906018 19001 consensus_queue.cc:1035] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b89cc6283342f6a509c8102abf3317" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:15.906244 19003 consensus_queue.cc:1035] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db898662bb384c3c858499d0016d37ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:09:15.907131 18977 tablet.cc:2369] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:15.916101 19007 mvcc.cc:205] Tried to move back new op lower bound from 7298853092979281920 to 7298853092809388032. Current Snapshot: MvccSnapshot[applied={T|T < 7298853092979281920}]
I20260620 09:09:15.916666 19008 mvcc.cc:205] Tried to move back new op lower bound from 7298853092979281920 to 7298853092809388032. Current Snapshot: MvccSnapshot[applied={T|T < 7298853092979281920}]
I20260620 09:09:15.923352 18516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:15.923611 18516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:15.926393 18516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e378ecc7e7a64042a1c6dcdccc6fc336: Sending DeleteTablet for 3 replicas of tablet e1e870b40ef44f6793b6c11285043abd
I20260620 09:09:15.927335 18776 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e870b40ef44f6793b6c11285043abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53914
I20260620 09:09:15.927484 18910 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e870b40ef44f6793b6c11285043abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49600
I20260620 09:09:15.927856 19016 tablet_replica.cc:330] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: stopping tablet replica
I20260620 09:09:15.927982 19017 tablet_replica.cc:330] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: stopping tablet replica
I20260620 09:09:15.928002 18642 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e870b40ef44f6793b6c11285043abd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43308
I20260620 09:09:15.928295 19017 raft_consensus.cc:2229] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:15.928296 19016 raft_consensus.cc:2229] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:15.928654 19017 raft_consensus.cc:2258] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:15.928691 19016 raft_consensus.cc:2258] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:15.929152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18579
I20260620 09:09:15.931159 19016 ts_tablet_manager.cc:1905] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:15.933769 19016 ts_tablet_manager.cc:1918] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:15.933887 19016 log.cc:1196] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/e1e870b40ef44f6793b6c11285043abd
I20260620 09:09:15.934171 19016 ts_tablet_manager.cc:1939] T e1e870b40ef44f6793b6c11285043abd P db898662bb384c3c858499d0016d37ad: Deleting consensus metadata
I20260620 09:09:15.935319 18504 catalog_manager.cc:4841] TS db898662bb384c3c858499d0016d37ad (127.0.98.2:34749): tablet e1e870b40ef44f6793b6c11285043abd (table test-table [id=54d65afb247a408e966c2828de7a6224]) successfully deleted
W20260620 09:09:15.937055 18502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34611 (error 108)
I20260620 09:09:15.937179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18713
I20260620 09:09:15.938746 19017 ts_tablet_manager.cc:1905] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:15.938829 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:15.938517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34611 (local address 127.0.0.1:43320)
0620 09:09:15.938663 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:15.938732 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 09:09:15.938979 18502 catalog_manager.cc:4584] TS 74b89cc6283342f6a509c8102abf3317 (127.0.98.1:34611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e870b40ef44f6793b6c11285043abd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34611: connect: Connection refused (error 111)
I20260620 09:09:15.940949 19017 ts_tablet_manager.cc:1918] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:15.941028 19017 log.cc:1196] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/e1e870b40ef44f6793b6c11285043abd
I20260620 09:09:15.941421 19017 ts_tablet_manager.cc:1939] T e1e870b40ef44f6793b6c11285043abd P cead037095cf4ec29fcff2deaf557bc7: Deleting consensus metadata
I20260620 09:09:15.942451 18502 catalog_manager.cc:4841] TS cead037095cf4ec29fcff2deaf557bc7 (127.0.98.3:35205): tablet e1e870b40ef44f6793b6c11285043abd (table test-table [id=54d65afb247a408e966c2828de7a6224]) successfully deleted
I20260620 09:09:15.944453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18847
I20260620 09:09:15.950415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 18485
2026-06-20T09:09:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2457 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T09:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:15Z Disabled control of system clock
I20260620 09:09:15.980650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38607
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:16.094182 19023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:16.094455 19023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:16.094491 19023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:16.098518 19023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:16.098579 19023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:16.098594 19023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:16.098611 19023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:16.098627 19023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:16.103358 19023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:16.104406 19023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:16.105865 19023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:16.114444 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.114463 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:16.114723 19023 server_base.cc:1034] running on GCE node
W20260620 09:09:16.114473 19031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:16.115494 19023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:16.116780 19023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:16.117957 19023 hybrid_clock.cc:648] HybridClock initialized: now 1781946556117967 us; error 76 us; skew 500 ppm
I20260620 09:09:16.119978 19023 webserver.cc:460] Webserver started at http://127.0.98.62:43115/ using document root <none> and password file <none>
I20260620 09:09:16.120579 19023 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:16.120641 19023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:16.120815 19023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:16.123211 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "de201df5d202705887b31be83131e0d6"
server_key_iv: "bf84cddb75908802ba1f4fa4d4662b24"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.123756 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "de201df5d202705887b31be83131e0d6"
server_key_iv: "bf84cddb75908802ba1f4fa4d4662b24"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.127903 19023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:16.130720 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.132691 19023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:16.132833 19023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "de201df5d202705887b31be83131e0d6"
server_key_iv: "bf84cddb75908802ba1f4fa4d4662b24"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.132993 19023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:16.157577 19023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:16.158362 19023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:16.158588 19023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:16.165683 19023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38607
I20260620 09:09:16.165694 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38607 every 8 connection(s)
I20260620 09:09:16.166728 19023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:16.169961 19090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:16.175607 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: Bootstrap starting.
I20260620 09:09:16.176978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19023
I20260620 09:09:16.177163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:16.177475 392 external_mini_cluster.cc:1428] Setting key f40a37dff8285a72ad9931c21b1bcafc
I20260620 09:09:16.178390 19090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:16.179270 19090 log.cc:824] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:16.181391 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: No bootstrap required, opened a new log
I20260620 09:09:16.184782 19090 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260620 09:09:16.185021 19090 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:16.185128 19090 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11af7ac41cf4e1fb272ecd1fa6bbec9, State: Initialized, Role: FOLLOWER
I20260620 09:09:16.185556 19090 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260620 09:09:16.185714 19090 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:16.185779 19090 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:16.185885 19090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:16.186817 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.178604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.98.62:38607)
0620 09:09:16.179313 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.179331 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.179398 (+ 67us) server_negotiation.cc:407] Connection header received
0620 09:09:16.180256 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.180277 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.180628 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.180994 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:16.181489 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.182448 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.183347 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.183829 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.184013 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.184875 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:16.184881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:16.184894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:16.185441 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:16.185511 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:16.185533 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:16.185659 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.185793 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.186101 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
I20260620 09:09:16.186818 19090 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260620 09:09:16.187258 19090 leader_election.cc:302] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11af7ac41cf4e1fb272ecd1fa6bbec9; no voters:
I20260620 09:09:16.187649 19090 leader_election.cc:288] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:16.187726 19095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:16.189087 19090 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:16.187958 19095 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [term 1 LEADER]: Becoming Leader. State: Replica: d11af7ac41cf4e1fb272ecd1fa6bbec9, State: Running, Role: LEADER
I20260620 09:09:16.189419 19095 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } }
I20260620 09:09:16.191316 19097 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } } }
I20260620 09:09:16.191316 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11af7ac41cf4e1fb272ecd1fa6bbec9. Latest consensus state: current_term: 1 leader_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11af7ac41cf4e1fb272ecd1fa6bbec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38607 } } }
I20260620 09:09:16.191828 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:16.191828 19097 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9 [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:16.194259 19109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:16.194344 19109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:16.194451 19110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:16.195436 19110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:16.201191 19110 catalog_manager.cc:1347] Generated new cluster ID: 12f24f43272b498b86b3a8d7d3f4ba51
I20260620 09:09:16.201277 19110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:16.221079 19110 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:16.222390 19110 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:16.228456 19110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: Generated new TSK 0
I20260620 09:09:16.229364 19110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:16.240914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--builtin_ntp_servers=127.0.98.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:16.357481 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:16.357770 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:16.357812 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:16.361888 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:16.361948 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:16.362006 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:16.366900 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:16.367975 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:16.369366 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:16.378926 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.378954 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.378965 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:16.379033 19114 server_base.cc:1034] running on GCE node
I20260620 09:09:16.379619 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:16.380260 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:16.381413 19114 hybrid_clock.cc:648] HybridClock initialized: now 1781946556381380 us; error 45 us; skew 500 ppm
I20260620 09:09:16.383323 19114 webserver.cc:460] Webserver started at http://127.0.98.1:40289/ using document root <none> and password file <none>
I20260620 09:09:16.383922 19114 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:16.384016 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:16.384255 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:16.386466 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "e677ba87ca064826b53d5b4715f572aa"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "ab63c7481f0da6196aaa76c386d0974a"
server_key_iv: "c9bf55743d8a73c8a7910d599c0cfbd6"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.387066 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e677ba87ca064826b53d5b4715f572aa"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "ab63c7481f0da6196aaa76c386d0974a"
server_key_iv: "c9bf55743d8a73c8a7910d599c0cfbd6"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.391243 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:16.393806 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.395051 19114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:16.395190 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "e677ba87ca064826b53d5b4715f572aa"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "ab63c7481f0da6196aaa76c386d0974a"
server_key_iv: "c9bf55743d8a73c8a7910d599c0cfbd6"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.395313 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:16.409196 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:16.410000 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:16.410213 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:16.410841 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:16.411828 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:16.411904 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.411978 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:16.412029 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.421337 19114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44395
I20260620 09:09:16.421401 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44395 every 8 connection(s)
I20260620 09:09:16.422366 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:16.427805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19114
I20260620 09:09:16.428009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:16.428404 392 external_mini_cluster.cc:1428] Setting key 8149ed6235278c3340805ce9acfabd60
I20260620 09:09:16.430739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--builtin_ntp_servers=127.0.98.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:16.433115 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.424305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36705 (local address 127.0.98.62:38607)
0620 09:09:16.424505 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.424510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.425691 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 09:09:16.427077 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.427080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.427131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.427214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:16.428227 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.429023 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.430707 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.431105 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.431233 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.432556 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:16.432559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:16.432562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:16.432575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:16.432594 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:16.432603 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:16.432664 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.432869 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.432936 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 09:09:16.434224 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.424787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.1:36705)
0620 09:09:16.425533 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.425576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.426835 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.427390 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.427397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.427498 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:16.428050 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.428068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.429175 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.429178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.430463 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.430475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.431281 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.431285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.431442 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.432210 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:16.432213 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:16.432279 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:16.432380 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:16.432691 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:16.432703 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:16.432712 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.432860 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.433355 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":113,"threads_started":1}
I20260620 09:09:16.435542 19242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260620 09:09:16.435835 19242 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:16.436584 19242 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260620 09:09:16.438530 19054 ts_manager.cc:194] Registered new tserver with Master: e677ba87ca064826b53d5b4715f572aa (127.0.98.1:44395)
I20260620 09:09:16.439806 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36705
W20260620 09:09:16.547107 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:16.547381 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:16.547473 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:16.551590 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:16.551663 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:16.551723 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:16.556697 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:16.557920 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:16.559435 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:16.569326 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.569334 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.569550 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:16.569684 19245 server_base.cc:1034] running on GCE node
I20260620 09:09:16.570161 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:16.570840 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:16.572044 19245 hybrid_clock.cc:648] HybridClock initialized: now 1781946556572016 us; error 44 us; skew 500 ppm
I20260620 09:09:16.574023 19245 webserver.cc:460] Webserver started at http://127.0.98.2:42721/ using document root <none> and password file <none>
I20260620 09:09:16.574626 19245 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:16.574723 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:16.574967 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:16.577124 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6644c72c9b04038b0e500758cbb38ae"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "beb7b125afef65e63418d1dd49db2cd0"
server_key_iv: "2aa3d32a804c622316786863ca44c081"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.577713 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6644c72c9b04038b0e500758cbb38ae"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "beb7b125afef65e63418d1dd49db2cd0"
server_key_iv: "2aa3d32a804c622316786863ca44c081"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.581535 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 09:09:16.584024 19259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.585417 19245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:16.585569 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "c6644c72c9b04038b0e500758cbb38ae"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "beb7b125afef65e63418d1dd49db2cd0"
server_key_iv: "2aa3d32a804c622316786863ca44c081"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.585705 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:16.599525 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:16.600291 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:16.600504 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:16.601186 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:16.602169 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:16.602240 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.602312 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:16.602356 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.611294 19245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42255
I20260620 09:09:16.611325 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42255 every 8 connection(s)
I20260620 09:09:16.612243 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:16.617892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19245
I20260620 09:09:16.617995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:09:16.618325 392 external_mini_cluster.cc:1428] Setting key 949d9b0f85c54fcc1e32fbf763f106fa
I20260620 09:09:16.620675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--builtin_ntp_servers=127.0.98.20:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:16.622736 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.614264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32993 (local address 127.0.98.62:38607)
0620 09:09:16.614425 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.614428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.615508 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 09:09:16.617017 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.617020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.617109 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.617186 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:16.618188 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.618815 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.620383 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.620740 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.620866 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.622273 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:16.622276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:16.622278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:16.622290 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:16.622309 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:16.622317 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:16.622372 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.622566 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.622626 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:09:16.623817 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.2:32993)
0620 09:09:16.615351 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.615393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.616779 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.617368 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.617376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.617470 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:16.617999 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.618022 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.618960 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.618963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.620248 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.620259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.621015 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.621018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.621218 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.621904 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:16.621908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:16.621978 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:16.622088 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:16.622399 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:16.622409 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:16.622418 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.622564 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.622900 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260620 09:09:16.625226 19373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260620 09:09:16.625492 19373 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:16.626065 19373 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260620 09:09:16.627344 19054 ts_manager.cc:194] Registered new tserver with Master: c6644c72c9b04038b0e500758cbb38ae (127.0.98.2:42255)
I20260620 09:09:16.627962 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32993
W20260620 09:09:16.740710 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:16.741024 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:16.741112 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:16.745226 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:16.745309 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:16.745396 19376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:16.750479 19376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:16.751713 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:16.753221 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:16.763657 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.763615 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:16.763768 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:16.763742 19376 server_base.cc:1034] running on GCE node
I20260620 09:09:16.764381 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:16.765125 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:16.766321 19376 hybrid_clock.cc:648] HybridClock initialized: now 1781946556766284 us; error 48 us; skew 500 ppm
I20260620 09:09:16.768234 19376 webserver.cc:460] Webserver started at http://127.0.98.3:42793/ using document root <none> and password file <none>
I20260620 09:09:16.768833 19376 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:16.768899 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:16.769233 19376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:16.771374 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b04df0eaa2e44c38b3c9ba3945993d4"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "bfb67378ce3f93891bd9b5578f37a2ed"
server_key_iv: "35cfca8df8a2277f1fd49d718ce61f4b"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.771962 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b04df0eaa2e44c38b3c9ba3945993d4"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "bfb67378ce3f93891bd9b5578f37a2ed"
server_key_iv: "35cfca8df8a2277f1fd49d718ce61f4b"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.775763 19376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:16.778288 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.779554 19376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:16.779673 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "5b04df0eaa2e44c38b3c9ba3945993d4"
format_stamp: "Formatted at 2026-06-20 09:09:16 on dist-test-slave-92t4"
server_key: "bfb67378ce3f93891bd9b5578f37a2ed"
server_key_iv: "35cfca8df8a2277f1fd49d718ce61f4b"
server_key_version: "encryptionkey@0"
I20260620 09:09:16.779785 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:16.802820 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:16.803591 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:16.803802 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:16.804440 19376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:16.805516 19376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:16.805588 19376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.805661 19376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:16.805704 19376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:16.814496 19376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37697
I20260620 09:09:16.814535 19503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37697 every 8 connection(s)
I20260620 09:09:16.815475 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:16.818357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19376
I20260620 09:09:16.818480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:16.818831 392 external_mini_cluster.cc:1428] Setting key 959c5952e415b9a331f39f7da51d88c7
I20260620 09:09:16.825726 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.817475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33027 (local address 127.0.98.62:38607)
0620 09:09:16.817626 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.817630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.818856 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 09:09:16.820596 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.820599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.820657 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.820737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:16.821827 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.822383 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.823581 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.823819 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.823936 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.825203 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:16.825206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:16.825208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:16.825222 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:16.825241 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:16.825249 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:16.825332 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.825504 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.825574 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:09:16.826714 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.817904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38607 (local address 127.0.98.3:33027)
0620 09:09:16.818690 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.818734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.820368 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.820971 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.820979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.821139 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:16.821679 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.821693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.822519 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.822524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.823450 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.823460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.823982 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.823986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.824173 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.824816 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:16.824818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:16.824877 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:16.824948 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:16.825332 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:16.825345 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:16.825353 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.825496 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.825806 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":159,"threads_started":1}
I20260620 09:09:16.828162 19504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38607
I20260620 09:09:16.828430 19504 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:16.829010 19504 heartbeater.cc:511] Master 127.0.98.62:38607 requested a full tablet report, sending...
I20260620 09:09:16.830143 19054 ts_manager.cc:194] Registered new tserver with Master: 5b04df0eaa2e44c38b3c9ba3945993d4 (127.0.98.3:37697)
I20260620 09:09:16.830685 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33027
I20260620 09:09:16.832983 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:16.839439 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.835316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.0.98.62:38607)
0620 09:09:16.835451 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.835455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.835609 (+ 154us) server_negotiation.cc:407] Connection header received
0620 09:09:16.835732 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.835734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.835774 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.835835 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:16.836203 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.836719 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.837671 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.837914 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.837993 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.838952 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:16.838955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:16.838958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:16.838971 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:16.838990 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:16.838997 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:16.839102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.839278 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.839317 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:09:16.842818 19054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:16.845948 19054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:16.860970 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.854768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:59330)
0620 09:09:16.855614 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.855664 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.855793 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.856688 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.856691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.856711 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:16.856848 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.856855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.857995 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.858001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.859537 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.859546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.860578 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.860581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.860679 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.860699 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.860763 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.860814 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":49,"thread_start_us":117,"threads_started":1}
I20260620 09:09:16.861091 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.854924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.0.98.1:44395)
0620 09:09:16.856257 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.856267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.856287 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:16.856351 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.856356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.856543 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.856992 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:16.857166 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.857883 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.859674 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.860438 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.860537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.860695 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.860790 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.860892 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":139,"threads_started":1}
I20260620 09:09:16.861428 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.854785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37697 (local address 127.0.0.1:57876)
0620 09:09:16.855881 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.855895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.855980 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.856571 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.856577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.856598 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:16.856744 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.856751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.858319 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.858323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.859343 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.859352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.860392 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.860395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.860475 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.860570 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.860679 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.860801 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260620 09:09:16.861438 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.855603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.98.3:37697)
0620 09:09:16.856063 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.856076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.856102 (+ 26us) server_negotiation.cc:407] Connection header received
0620 09:09:16.856185 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.856191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.856404 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.856562 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:16.856878 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.858174 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.859478 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.860264 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.860381 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.860605 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.860886 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.860984 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":104,"threads_started":1}
I20260620 09:09:16.862783 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.855084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42255 (local address 127.0.0.1:37880)
0620 09:09:16.856144 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:16.856157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:16.856238 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:16.856755 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:16.856757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:16.856769 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:16.856861 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.856865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.858274 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.858277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.859536 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:16.859546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.862392 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.862396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:16.862491 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.862513 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 09:09:16.862591 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:09:16.862661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":62,"thread_start_us":84,"threads_started":1}
I20260620 09:09:16.862900 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:16.855502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.98.2:42255)
0620 09:09:16.856063 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:16.856076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:16.856339 (+ 263us) server_negotiation.cc:407] Connection header received
0620 09:09:16.856407 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:16.856410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:16.856615 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:16.856764 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:16.856979 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.858152 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.860681 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:16.862060 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:16.862162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:16.862332 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:16.862648 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 09:09:16.862747 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":124,"threads_started":1}
I20260620 09:09:16.864208 19438 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552a92e4794373891081912a262576 (DEFAULT_TABLE table=test-table [id=f28e5d6c91344c8e88dadcb6cd4245c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:16.864217 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552a92e4794373891081912a262576 (DEFAULT_TABLE table=test-table [id=f28e5d6c91344c8e88dadcb6cd4245c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:16.865259 19438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552a92e4794373891081912a262576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:16.865278 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552a92e4794373891081912a262576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:16.865882 19307 tablet_service.cc:1467] Processing CreateTablet for tablet 2a552a92e4794373891081912a262576 (DEFAULT_TABLE table=test-table [id=f28e5d6c91344c8e88dadcb6cd4245c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:16.866824 19307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a552a92e4794373891081912a262576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:16.871534 19521 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Bootstrap starting.
I20260620 09:09:16.873286 19522 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Bootstrap starting.
I20260620 09:09:16.873503 19523 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Bootstrap starting.
I20260620 09:09:16.874116 19521 tablet_bootstrap.cc:654] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:16.875007 19521 log.cc:824] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:16.875275 19522 tablet_bootstrap.cc:654] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:16.875582 19523 tablet_bootstrap.cc:654] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:16.876072 19522 log.cc:824] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:16.876367 19523 log.cc:824] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:16.877426 19521 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: No bootstrap required, opened a new log
I20260620 09:09:16.877627 19521 ts_tablet_manager.cc:1397] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:16.878270 19523 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: No bootstrap required, opened a new log
I20260620 09:09:16.878273 19522 tablet_bootstrap.cc:492] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: No bootstrap required, opened a new log
I20260620 09:09:16.878481 19523 ts_tablet_manager.cc:1397] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:09:16.878480 19522 ts_tablet_manager.cc:1397] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 09:09:16.881840 19523 raft_consensus.cc:348] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.882154 19523 raft_consensus.cc:374] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:16.882210 19523 raft_consensus.cc:729] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6644c72c9b04038b0e500758cbb38ae, State: Initialized, Role: FOLLOWER
I20260620 09:09:16.882221 19521 raft_consensus.cc:348] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.882483 19521 raft_consensus.cc:374] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:16.882540 19521 raft_consensus.cc:729] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b04df0eaa2e44c38b3c9ba3945993d4, State: Initialized, Role: FOLLOWER
I20260620 09:09:16.882705 19523 consensus_queue.cc:260] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.882690 19522 raft_consensus.cc:348] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.882921 19522 raft_consensus.cc:374] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:16.882968 19522 raft_consensus.cc:729] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677ba87ca064826b53d5b4715f572aa, State: Initialized, Role: FOLLOWER
I20260620 09:09:16.882967 19521 consensus_queue.cc:260] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.883536 19373 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260620 09:09:16.883949 19521 ts_tablet_manager.cc:1428] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:16.883973 19523 ts_tablet_manager.cc:1428] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260620 09:09:16.884313 19504 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260620 09:09:16.884539 19522 consensus_queue.cc:260] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:16.885231 19242 heartbeater.cc:503] Master 127.0.98.62:38607 was elected leader, sending a full tablet report...
I20260620 09:09:16.885902 19522 ts_tablet_manager.cc:1428] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 09:09:16.923641 19243 tablet.cc:2369] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:17.013451 19527 raft_consensus.cc:482] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:17.013643 19527 raft_consensus.cc:504] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:17.014744 19527 leader_election.cc:288] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e677ba87ca064826b53d5b4715f572aa (127.0.98.1:44395), 5b04df0eaa2e44c38b3c9ba3945993d4 (127.0.98.3:37697)
I20260620 09:09:17.018769 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.015100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34761 (local address 127.0.98.1:44395)
0620 09:09:17.015250 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:17.015256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:17.015276 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:17.015451 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:17.015455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:17.015517 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:17.015621 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:17.015941 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.016571 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.017756 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.018178 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.018261 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:17.018339 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:17.018553 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 09:09:17.018634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 09:09:17.018769 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.014991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.98.2:34761)
0620 09:09:17.015201 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:17.015219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:17.015323 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:17.015661 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:17.015664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:17.015690 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:17.015816 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:17.015827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.016691 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.016694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:17.017624 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:17.017634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.018307 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.018313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:17.018413 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:17.018443 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 09:09:17.018514 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:09:17.018605 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2}
I20260620 09:09:17.018811 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.015104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42005 (local address 127.0.98.3:37697)
0620 09:09:17.015258 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:17.015264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:17.015645 (+ 381us) server_negotiation.cc:407] Connection header received
0620 09:09:17.015742 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:17.015745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:17.015798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:17.015886 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:17.016186 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.016709 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.017827 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.018249 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.018358 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:17.018417 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:17.018624 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 09:09:17.018703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 09:09:17.018786 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.014991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37697 (local address 127.0.98.2:42005)
0620 09:09:17.015527 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:17.015542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:17.015641 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:17.015927 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:17.015930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:17.015946 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:17.016078 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:17.016084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.016820 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.016823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:17.017667 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:17.017676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.018378 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.018381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:17.018469 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:17.018489 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 09:09:17.018586 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 09:09:17.018649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":166,"thread_start_us":123,"threads_started":1}
I20260620 09:09:17.019536 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552a92e4794373891081912a262576" candidate_uuid: "c6644c72c9b04038b0e500758cbb38ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677ba87ca064826b53d5b4715f572aa" is_pre_election: true
I20260620 09:09:17.019546 19458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552a92e4794373891081912a262576" candidate_uuid: "c6644c72c9b04038b0e500758cbb38ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" is_pre_election: true
I20260620 09:09:17.019888 19196 raft_consensus.cc:2454] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6644c72c9b04038b0e500758cbb38ae in term 0.
I20260620 09:09:17.019888 19458 raft_consensus.cc:2454] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6644c72c9b04038b0e500758cbb38ae in term 0.
I20260620 09:09:17.020452 19262 leader_election.cc:302] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b04df0eaa2e44c38b3c9ba3945993d4, c6644c72c9b04038b0e500758cbb38ae, e677ba87ca064826b53d5b4715f572aa; no voters:
I20260620 09:09:17.020782 19527 raft_consensus.cc:2781] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:17.020890 19527 raft_consensus.cc:482] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:17.021009 19527 raft_consensus.cc:3037] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:17.022109 19527 raft_consensus.cc:504] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:17.022492 19527 leader_election.cc:288] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [CANDIDATE]: Term 1 election: Requested vote from peers e677ba87ca064826b53d5b4715f572aa (127.0.98.1:44395), 5b04df0eaa2e44c38b3c9ba3945993d4 (127.0.98.3:37697)
I20260620 09:09:17.022877 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552a92e4794373891081912a262576" candidate_uuid: "c6644c72c9b04038b0e500758cbb38ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e677ba87ca064826b53d5b4715f572aa"
I20260620 09:09:17.023052 19196 raft_consensus.cc:3037] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:17.024044 19196 raft_consensus.cc:2454] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6644c72c9b04038b0e500758cbb38ae in term 1.
I20260620 09:09:17.024322 19458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a552a92e4794373891081912a262576" candidate_uuid: "c6644c72c9b04038b0e500758cbb38ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4"
I20260620 09:09:17.024391 19263 leader_election.cc:302] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6644c72c9b04038b0e500758cbb38ae, e677ba87ca064826b53d5b4715f572aa; no voters:
I20260620 09:09:17.024466 19458 raft_consensus.cc:3037] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:17.024607 19527 raft_consensus.cc:2781] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:17.024838 19527 raft_consensus.cc:686] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 1 LEADER]: Becoming Leader. State: Replica: c6644c72c9b04038b0e500758cbb38ae, State: Running, Role: LEADER
I20260620 09:09:17.025249 19527 consensus_queue.cc:237] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } }
I20260620 09:09:17.025851 19458 raft_consensus.cc:2454] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6644c72c9b04038b0e500758cbb38ae in term 1.
I20260620 09:09:17.028855 19054 catalog_manager.cc:5495] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae reported cstate change: term changed from 0 to 1, leader changed from <none> to c6644c72c9b04038b0e500758cbb38ae (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6644c72c9b04038b0e500758cbb38ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6644c72c9b04038b0e500758cbb38ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42255 } health_report { overall_health: HEALTHY } } }
I20260620 09:09:17.057601 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.053447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.98.2:42255)
0620 09:09:17.053629 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:17.053633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:17.053647 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:09:17.053719 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:17.053721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:17.053776 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:17.053924 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:17.054181 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.054696 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.056054 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:17.056323 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:17.056407 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:17.056649 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:17.057149 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:17.057271 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:17.057354 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 09:09:17.057387 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 09:09:17.063724 19458 raft_consensus.cc:1261] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 1 FOLLOWER]: Refusing update from remote peer c6644c72c9b04038b0e500758cbb38ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:17.063724 19196 raft_consensus.cc:1261] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 1 FOLLOWER]: Refusing update from remote peer c6644c72c9b04038b0e500758cbb38ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:17.064517 19527 consensus_queue.cc:1035] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e677ba87ca064826b53d5b4715f572aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:17.064742 19531 consensus_queue.cc:1035] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b04df0eaa2e44c38b3c9ba3945993d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 09:09:17.066411 19505 tablet.cc:2369] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:17.073310 19536 mvcc.cc:205] Tried to move back new op lower bound from 7298853097725136896 to 7298853097578885120. Current Snapshot: MvccSnapshot[applied={T|T < 7298853097725136896}]
I20260620 09:09:17.075556 19535 mvcc.cc:205] Tried to move back new op lower bound from 7298853097725136896 to 7298853097578885120. Current Snapshot: MvccSnapshot[applied={T|T < 7298853097725136896}]
I20260620 09:09:17.084599 19052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:17.084770 19052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:17.087425 19052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11af7ac41cf4e1fb272ecd1fa6bbec9: Sending DeleteTablet for 3 replicas of tablet 2a552a92e4794373891081912a262576
I20260620 09:09:17.088277 19438 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a552a92e4794373891081912a262576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:17 UTC) from {username='slave'} at 127.0.0.1:57876
I20260620 09:09:17.088284 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a552a92e4794373891081912a262576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:17 UTC) from {username='slave'} at 127.0.0.1:59330
I20260620 09:09:17.088640 19307 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a552a92e4794373891081912a262576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:17 UTC) from {username='slave'} at 127.0.0.1:37880
I20260620 09:09:17.088882 19545 tablet_replica.cc:330] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: stopping tablet replica
I20260620 09:09:17.089082 19546 tablet_replica.cc:330] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: stopping tablet replica
I20260620 09:09:17.089381 19544 tablet_replica.cc:330] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa: stopping tablet replica
I20260620 09:09:17.089697 19544 raft_consensus.cc:2229] T 2a552a92e4794373891081912a262576 P e677ba87ca064826b53d5b4715f572aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:17.089900 19546 raft_consensus.cc:2229] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:17.089967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19114
I20260620 09:09:17.090690 19545 raft_consensus.cc:2229] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:17.091048 19545 raft_consensus.cc:2258] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:17.092715 19545 ts_tablet_manager.cc:1905] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:17.095144 19545 ts_tablet_manager.cc:1918] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:17.095264 19545 log.cc:1196] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/2a552a92e4794373891081912a262576
I20260620 09:09:17.095585 19545 ts_tablet_manager.cc:1939] T 2a552a92e4794373891081912a262576 P 5b04df0eaa2e44c38b3c9ba3945993d4: Deleting consensus metadata
I20260620 09:09:17.096731 19040 catalog_manager.cc:4841] TS 5b04df0eaa2e44c38b3c9ba3945993d4 (127.0.98.3:37697): tablet 2a552a92e4794373891081912a262576 (table test-table [id=f28e5d6c91344c8e88dadcb6cd4245c5]) successfully deleted
I20260620 09:09:17.097368 19546 raft_consensus.cc:2258] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 09:09:17.098420 19041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44395 (error 108)
I20260620 09:09:17.098894 19546 ts_tablet_manager.cc:1905] T 2a552a92e4794373891081912a262576 P c6644c72c9b04038b0e500758cbb38ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:17.099490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19245
I20260620 09:09:17.099570 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.099257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44395 (local address 127.0.0.1:59338)
0620 09:09:17.099408 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:17.099481 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 09:09:17.100935 19041 catalog_manager.cc:4584] TS e677ba87ca064826b53d5b4715f572aa (127.0.98.1:44395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a552a92e4794373891081912a262576: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44395: connect: Connection refused (error 111)
I20260620 09:09:17.106101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19376
I20260620 09:09:17.107580 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:17.106251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42255 (local address 127.0.0.1:37902)
0620 09:09:17.107407 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:17.107474 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1049}
I20260620 09:09:17.111718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19023
2026-06-20T09:09:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:17 dist-test-slave-92t4 krb5kdc[19550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:17 dist-test-slave-92t4 krb5kdc[19550](info): set up 2 sockets
Jun 20 09:09:17 dist-test-slave-92t4 krb5kdc[19550](info): commencing operation
krb5kdc: starting...
W20260620 09:09:18.145740 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:18.272014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:18.388901 19563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:18.389312 19563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:18.389403 19563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:18.393729 19563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:18.393836 19563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:18.393875 19563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:18.393930 19563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:18.393970 19563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:18.399762 19563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:18.401104 19563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:18.402293 19563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:18.411440 19563 server_base.cc:1034] running on GCE node
W20260620 09:09:18.411374 19571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:18.411370 19568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:18.411374 19569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:18.412247 19563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:18.413573 19563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:18.414762 19563 hybrid_clock.cc:648] HybridClock initialized: now 1781946558414729 us; error 49 us; skew 500 ppm
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:18.417733 19563 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:18.418787 19563 webserver.cc:460] Webserver started at http://127.0.98.62:36359/ using document root <none> and password file <none>
I20260620 09:09:18.419445 19563 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:18.419544 19563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:18.419803 19563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:18.422103 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "7c21cd74163a47ff8977f7c80fd52e4e"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "c2ae590c653a440c560f604b77ece002"
server_key_iv: "f5991b70653a4ce42a61154f4e0a5626"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.422714 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c21cd74163a47ff8977f7c80fd52e4e"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "c2ae590c653a440c560f604b77ece002"
server_key_iv: "f5991b70653a4ce42a61154f4e0a5626"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.426872 19563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:18.429590 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.430888 19563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:18.431051 19563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "7c21cd74163a47ff8977f7c80fd52e4e"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "c2ae590c653a440c560f604b77ece002"
server_key_iv: "f5991b70653a4ce42a61154f4e0a5626"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.431186 19563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:18.450526 19563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:18.453923 19563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:18.454149 19563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:18.461602 19563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40917
I20260620 09:09:18.461618 19630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40917 every 8 connection(s)
I20260620 09:09:18.462606 19563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:18.465480 19631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:18.469017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19563
I20260620 09:09:18.469220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:18.469513 392 external_mini_cluster.cc:1428] Setting key e88473264f106e267c254a615dc6ca28
I20260620 09:09:18.471509 19631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: Bootstrap starting.
I20260620 09:09:18.474324 19631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:18.475205 19631 log.cc:824] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:18.477604 19631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: No bootstrap required, opened a new log
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:18.482604 19631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260620 09:09:18.482888 19631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:18.482949 19631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c21cd74163a47ff8977f7c80fd52e4e, State: Initialized, Role: FOLLOWER
I20260620 09:09:18.483471 19631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260620 09:09:18.483599 19631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:18.483685 19631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:18.483805 19631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:18.484794 19631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260620 09:09:18.485211 19631 leader_election.cc:302] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c21cd74163a47ff8977f7c80fd52e4e; no voters:
I20260620 09:09:18.485579 19631 leader_election.cc:288] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:18.485656 19636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:18.485862 19636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [term 1 LEADER]: Becoming Leader. State: Replica: 7c21cd74163a47ff8977f7c80fd52e4e, State: Running, Role: LEADER
I20260620 09:09:18.486049 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:18.470971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.98.62:40917)
0620 09:09:18.471414 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:18.471441 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:18.471485 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:09:18.472303 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:18.472325 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:18.472763 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:18.473185 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:18.473872 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.475391 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.476509 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.477239 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.477404 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:18.480281 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:18.480290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:18.480310 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:18.480348 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:18.482533 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.483080 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.483085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.483090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:18.483216 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.483525 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.483529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.483531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:18.484129 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:18.484347 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:18.484651 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 09:09:18.485006 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260620 09:09:18.486171 19636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } }
I20260620 09:09:18.486953 19631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:18.488765 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } } }
I20260620 09:09:18.488902 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:18.489240 19638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c21cd74163a47ff8977f7c80fd52e4e. Latest consensus state: current_term: 1 leader_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c21cd74163a47ff8977f7c80fd52e4e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40917 } } }
I20260620 09:09:18.489351 19651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:18.489760 19638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:18.492879 19651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:18.498584 19651 catalog_manager.cc:1347] Generated new cluster ID: 886f84648d7f447a865dfea76f5eed0d
I20260620 09:09:18.498654 19651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:18.508648 19651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:18.509696 19651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:18.517905 19651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: Generated new TSK 0
I20260620 09:09:18.518709 19651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 09:09:18.551616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:18.688393 19657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:18.688694 19657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:18.688740 19657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:18.692929 19657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:18.693007 19657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:18.693087 19657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:18.698565 19657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:18.699707 19657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:18.700744 19657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:18.710441 19663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:18.710445 19665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:18.710557 19657 server_base.cc:1034] running on GCE node
W20260620 09:09:18.710426 19662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:18.711313 19657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:18.711905 19657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:18.713137 19657 hybrid_clock.cc:648] HybridClock initialized: now 1781946558713118 us; error 29 us; skew 500 ppm
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:18.716183 19657 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 09:09:18.717317 19657 webserver.cc:460] Webserver started at http://127.0.98.1:38085/ using document root <none> and password file <none>
I20260620 09:09:18.717959 19657 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:18.718026 19657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:18.718283 19657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:18.720554 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce7b651930504c4bac4623d7074aee9f"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "5f29204414a45ee2e72e4dc2db67953d"
server_key_iv: "56349a0c770322d487bcfe6f0f08c00e"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.721257 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce7b651930504c4bac4623d7074aee9f"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "5f29204414a45ee2e72e4dc2db67953d"
server_key_iv: "56349a0c770322d487bcfe6f0f08c00e"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.725370 19657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 09:09:18.727955 19672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.729290 19657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:18.729423 19657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "ce7b651930504c4bac4623d7074aee9f"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "5f29204414a45ee2e72e4dc2db67953d"
server_key_iv: "56349a0c770322d487bcfe6f0f08c00e"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.729543 19657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:18.745303 19657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:18.748328 19657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:18.748541 19657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:18.749229 19657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:18.750095 19657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:18.750170 19657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.750244 19657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:18.750298 19657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.759680 19657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44223
I20260620 09:09:18.759701 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44223 every 8 connection(s)
I20260620 09:09:18.760687 19657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:18.769455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19657
I20260620 09:09:18.769615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:18.770023 392 external_mini_cluster.cc:1428] Setting key 75030a6e3e8e74c8cd0467e8f14dbf17
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:18.776099 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:18.762693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35063 (local address 127.0.98.62:40917)
0620 09:09:18.762852 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:18.762858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:18.764131 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 09:09:18.765315 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:18.765318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:18.765367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:18.765454 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:18.766667 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.767201 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.768418 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.768656 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.768745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:18.771936 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:18.771939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:18.771941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:18.771963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:18.773756 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.774289 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.774296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.774301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:18.774432 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.774776 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.774782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.774787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:18.775064 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:18.775172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:18.775690 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 09:09:18.775845 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 09:09:18.776911 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:18.763109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40917 (local address 127.0.98.1:35063)
0620 09:09:18.763950 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:18.764009 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:18.765070 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:18.765616 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:18.765623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:18.766034 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:18.766540 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:18.766553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.767332 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.767335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:18.768282 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:18.768291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.768796 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.768798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:18.768970 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:18.769792 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:18.769817 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:18.771774 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:18.773891 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:18.773897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:18.773910 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:18.774184 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:18.774532 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:18.774535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:18.774536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:18.774675 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:18.775170 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:18.775175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:18.775428 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 09:09:18.775605 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 09:09:18.775878 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":125,"threads_started":1}
I20260620 09:09:18.778301 19786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40917
I20260620 09:09:18.778599 19786 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:18.779263 19786 heartbeater.cc:511] Master 127.0.98.62:40917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 09:09:18.781281 19595 ts_manager.cc:194] Registered new tserver with Master: ce7b651930504c4bac4623d7074aee9f (127.0.98.1:44223)
I20260620 09:09:18.782465 19595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 09:09:18.800418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:18.915288 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:18.915568 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:18.915612 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:18.919847 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:18.919931 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:18.919996 19791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:18.925612 19791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:18.926777 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:18.927922 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:18.937953 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:18.937987 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:18.938016 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:18.938331 19791 server_base.cc:1034] running on GCE node
I20260620 09:09:18.938788 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:18.939426 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:18.940618 19791 hybrid_clock.cc:648] HybridClock initialized: now 1781946558940584 us; error 48 us; skew 500 ppm
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:18.943557 19791 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 09:09:18.944422 19791 webserver.cc:460] Webserver started at http://127.0.98.2:33059/ using document root <none> and password file <none>
I20260620 09:09:18.945019 19791 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:18.945168 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:18.945407 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:18.947476 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "56441041aae649e390d65ec16ad9c1bd"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "dc01befa9ad3112de959c73d73d9ff2e"
server_key_iv: "1f0897cf30f7f5e9c3685baf5da56a04"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.948083 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56441041aae649e390d65ec16ad9c1bd"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "dc01befa9ad3112de959c73d73d9ff2e"
server_key_iv: "1f0897cf30f7f5e9c3685baf5da56a04"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.951876 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:09:18.954608 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.955941 19791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:18.956106 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "56441041aae649e390d65ec16ad9c1bd"
format_stamp: "Formatted at 2026-06-20 09:09:18 on dist-test-slave-92t4"
server_key: "dc01befa9ad3112de959c73d73d9ff2e"
server_key_iv: "1f0897cf30f7f5e9c3685baf5da56a04"
server_key_version: "encryptionkey@0"
I20260620 09:09:18.956233 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:18.969915 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:18.973433 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:18.973664 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:18.974289 19791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:18.975158 19791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:18.975229 19791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.975301 19791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:18.975353 19791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:18.984560 19791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35203
I20260620 09:09:18.984617 19919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35203 every 8 connection(s)
I20260620 09:09:18.985638 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:18.989348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19791
I20260620 09:09:18.989485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:09:18.989780 392 external_mini_cluster.cc:1428] Setting key f62b94d0b0f93b07c373ed1759f3d504
Jun 20 09:09:18 dist-test-slave-92t4 krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 09:09:19.001142 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:18.987609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59373 (local address 127.0.98.62:40917)
0620 09:09:18.987733 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:18.987736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:18.988693 (+ 957us) server_negotiation.cc:407] Connection header received
0620 09:09:18.990255 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:18.990259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:18.990330 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:18.990407 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:18.991687 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.992227 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.993496 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.993860 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.993986 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:18.996858 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:18.996864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:18.996867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:18.996898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:18.998666 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.999274 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.999278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.999280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:18.999375 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:18.999782 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:18.999785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:18.999788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:19.000063 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:19.000152 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.000801 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.000921 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:09:19.002261 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:18.988042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40917 (local address 127.0.98.2:59373)
0620 09:09:18.988538 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:18.988582 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:18.990040 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:18.990586 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:18.990593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:18.990987 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:18.991547 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:18.991561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.992375 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.992381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:18.993360 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:18.993370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:18.994032 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:18.994035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:18.994189 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:18.994820 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:18.994836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:18.996656 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:18.998814 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:18.998821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:18.998836 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:18.999153 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:18.999483 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:18.999485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:18.999486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:18.999670 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:19.000205 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:19.000211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:19.000508 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.000727 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.001177 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 09:09:19.003674 19920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40917
I20260620 09:09:19.003963 19920 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:19.004575 19920 heartbeater.cc:511] Master 127.0.98.62:40917 requested a full tablet report, sending...
I20260620 09:09:19.005797 19595 ts_manager.cc:194] Registered new tserver with Master: 56441041aae649e390d65ec16ad9c1bd (127.0.98.2:35203)
I20260620 09:09:19.006404 19595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 09:09:19.019665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:19.136320 19925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:19.136659 19925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:19.136754 19925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:19.141371 19925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:19.141469 19925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:19.141566 19925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:19.147141 19925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:19.148247 19925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:19.149355 19925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:19.159255 19930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:19.159252 19933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:19.159255 19931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:19.159624 19925 server_base.cc:1034] running on GCE node
I20260620 09:09:19.160145 19925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:19.160938 19925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:19.162142 19925 hybrid_clock.cc:648] HybridClock initialized: now 1781946559162122 us; error 33 us; skew 500 ppm
Jun 20 09:09:19 dist-test-slave-92t4 krb5kdc[19550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:19.165134 19925 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 09:09:19.166142 19925 webserver.cc:460] Webserver started at http://127.0.98.3:34543/ using document root <none> and password file <none>
I20260620 09:09:19.166742 19925 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:19.166832 19925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:19.167078 19925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:19.169298 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5056118e5344b59b6d55ca4f2fc999a"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "be905a6f69a0d48300b76ccf4dcdece5"
server_key_iv: "09559790f46c8dbf21138c158c1d421c"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.169888 19925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5056118e5344b59b6d55ca4f2fc999a"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "be905a6f69a0d48300b76ccf4dcdece5"
server_key_iv: "09559790f46c8dbf21138c158c1d421c"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.173933 19925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:09:19.176501 19940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:19.177838 19925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:19.177989 19925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "b5056118e5344b59b6d55ca4f2fc999a"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "be905a6f69a0d48300b76ccf4dcdece5"
server_key_iv: "09559790f46c8dbf21138c158c1d421c"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.178109 19925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:19.197830 19925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:19.201503 19925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:19.201731 19925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:19.202385 19925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:19.203246 19925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:19.203317 19925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:19.203389 19925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:19.203431 19925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:19.212113 19925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33767
I20260620 09:09:19.212136 20053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33767 every 8 connection(s)
I20260620 09:09:19.213068 19925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:19.216485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 19925
I20260620 09:09:19.216629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:19.216903 392 external_mini_cluster.cc:1428] Setting key 94ba7045438afea92a9d46e567e7c6cf
Jun 20 09:09:19 dist-test-slave-92t4 krb5kdc[19550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:19.226909 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.214815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46933 (local address 127.0.98.62:40917)
0620 09:09:19.214961 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.214965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.215776 (+ 811us) server_negotiation.cc:407] Connection header received
0620 09:09:19.216816 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.216819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.216880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.216997 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:19.218618 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.219293 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.220503 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.220713 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.220789 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.223235 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:19.223238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:19.223240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:19.223265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:19.224839 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:19.225414 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:19.225417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:19.225418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:19.225510 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:19.225815 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:19.225817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:19.225818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:19.225995 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:19.226069 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.226650 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.226773 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:19.227777 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.215177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40917 (local address 127.0.98.3:46933)
0620 09:09:19.215628 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.215670 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.216495 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.217205 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.217213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.217740 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:19.218482 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.218504 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.219412 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.219415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.220378 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.220387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.220843 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.220845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.220996 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.221553 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:19.221570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:19.223077 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:19.224954 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:19.224959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:19.224973 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:19.225309 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:19.225588 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:19.225591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:19.225592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:19.225730 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:19.226087 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:19.226093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:19.226381 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.226548 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.226818 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260620 09:09:19.228968 20054 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40917
I20260620 09:09:19.229270 20054 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:19.229882 20054 heartbeater.cc:511] Master 127.0.98.62:40917 requested a full tablet report, sending...
I20260620 09:09:19.231035 19594 ts_manager.cc:194] Registered new tserver with Master: b5056118e5344b59b6d55ca4f2fc999a (127.0.98.3:33767)
I20260620 09:09:19.231601 19594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46933
I20260620 09:09:19.243211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:19.252617 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.245345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.98.62:40917)
0620 09:09:19.245503 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.245511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.245650 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:09:19.245750 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.245752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.245798 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.245865 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:19.246546 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.247050 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.247868 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.248072 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.248144 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.249547 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:19.249550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:19.249552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:19.249576 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:19.250981 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:19.251351 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:19.251354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:19.251355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:19.251425 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:19.251693 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:19.251696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:19.251697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:19.251938 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:19.252062 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.252260 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.252373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:09:19.255908 19594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:19.259019 19594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:19.273888 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.268082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33767 (local address 127.0.0.1:39104)
0620 09:09:19.269105 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.269118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.269193 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.269869 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.269871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.269892 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:19.269951 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.269955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.270936 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.270942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.272394 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.272404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.273291 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.273295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.273409 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.273447 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.273573 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.273703 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":44,"thread_start_us":87,"threads_started":1}
I20260620 09:09:19.274040 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.268537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.0.98.2:35203)
0620 09:09:19.269159 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.269174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.269197 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:09:19.269276 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.269281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.269482 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.269629 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:19.269867 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.271017 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.272571 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.273454 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.273550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.273688 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.273774 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.273869 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":95,"threads_started":1}
I20260620 09:09:19.273900 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.268009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35203 (local address 127.0.0.1:46854)
0620 09:09:19.268831 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.268846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.268925 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.269631 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.269637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.269660 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:19.269749 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.269757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.271151 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.271154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.272430 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.272441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.273569 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.273572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.273652 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.273671 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.273742 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.273795 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":41,"thread_start_us":120,"threads_started":1}
I20260620 09:09:19.274562 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.268535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.98.3:33767)
0620 09:09:19.268929 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.268941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.269285 (+ 344us) server_negotiation.cc:407] Connection header received
0620 09:09:19.269369 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.269376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.269564 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.269702 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:19.270061 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.270795 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.272524 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.273174 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.273291 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.273463 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.274259 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.274375 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260620 09:09:19.274821 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.267720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44223 (local address 127.0.0.1:45694)
0620 09:09:19.268527 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.268579 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.268728 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.269434 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.269440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.269464 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:19.269625 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.269633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.272020 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.272023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.273164 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.273175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.274445 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.274447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.274512 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.274529 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.274625 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.274696 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260620 09:09:19.275055 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.267921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.0.98.1:44223)
0620 09:09:19.268467 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.268485 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.268880 (+ 395us) server_negotiation.cc:407] Connection header received
0620 09:09:19.269016 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.269023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.269292 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.269444 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:19.269760 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.271017 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.273308 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.274148 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.274249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.274384 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.274658 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.274757 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":90,"threads_started":1}
I20260620 09:09:19.277529 19854 tablet_service.cc:1467] Processing CreateTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 (DEFAULT_TABLE table=test-table [id=7d1f4bef821344f5a21065717dda3341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:19.277549 19720 tablet_service.cc:1467] Processing CreateTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 (DEFAULT_TABLE table=test-table [id=7d1f4bef821344f5a21065717dda3341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:19.277550 19988 tablet_service.cc:1467] Processing CreateTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 (DEFAULT_TABLE table=test-table [id=7d1f4bef821344f5a21065717dda3341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:19.278585 19854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fffb7ab78143d590f0e4ac115e0468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:19.278601 19720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fffb7ab78143d590f0e4ac115e0468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:19.278601 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fffb7ab78143d590f0e4ac115e0468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:19.286294 20071 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Bootstrap starting.
I20260620 09:09:19.286896 20073 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Bootstrap starting.
I20260620 09:09:19.288961 20073 tablet_bootstrap.cc:654] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:19.286773 20072 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Bootstrap starting.
I20260620 09:09:19.289407 20071 tablet_bootstrap.cc:654] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:19.290216 20073 log.cc:824] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:19.290261 20071 log.cc:824] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:19.291381 20072 tablet_bootstrap.cc:654] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:19.292219 20072 log.cc:824] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:19.292476 20071 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: No bootstrap required, opened a new log
I20260620 09:09:19.292510 20073 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: No bootstrap required, opened a new log
I20260620 09:09:19.292690 20073 ts_tablet_manager.cc:1397] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:19.292694 20071 ts_tablet_manager.cc:1397] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 09:09:19.294188 20072 tablet_bootstrap.cc:492] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: No bootstrap required, opened a new log
I20260620 09:09:19.294370 20072 ts_tablet_manager.cc:1397] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 09:09:19.296060 20071 raft_consensus.cc:348] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.296361 20071 raft_consensus.cc:374] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:19.296414 20071 raft_consensus.cc:729] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56441041aae649e390d65ec16ad9c1bd, State: Initialized, Role: FOLLOWER
I20260620 09:09:19.296736 20073 raft_consensus.cc:348] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.296873 20071 consensus_queue.cc:260] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.296996 20073 raft_consensus.cc:374] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:19.297070 20073 raft_consensus.cc:729] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5056118e5344b59b6d55ca4f2fc999a, State: Initialized, Role: FOLLOWER
I20260620 09:09:19.297667 19920 heartbeater.cc:503] Master 127.0.98.62:40917 was elected leader, sending a full tablet report...
I20260620 09:09:19.297960 20071 ts_tablet_manager.cc:1428] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 09:09:19.298132 20073 consensus_queue.cc:260] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.298815 20054 heartbeater.cc:503] Master 127.0.98.62:40917 was elected leader, sending a full tablet report...
I20260620 09:09:19.299146 20073 ts_tablet_manager.cc:1428] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 09:09:19.299028 20072 raft_consensus.cc:348] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.299268 20072 raft_consensus.cc:374] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:19.299314 20072 raft_consensus.cc:729] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b651930504c4bac4623d7074aee9f, State: Initialized, Role: FOLLOWER
I20260620 09:09:19.299768 20072 consensus_queue.cc:260] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.300464 19786 heartbeater.cc:503] Master 127.0.98.62:40917 was elected leader, sending a full tablet report...
I20260620 09:09:19.300745 20072 ts_tablet_manager.cc:1428] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:09:19.462065 20078 raft_consensus.cc:482] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:19.462320 20078 raft_consensus.cc:504] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.463682 20078 leader_election.cc:288] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce7b651930504c4bac4623d7074aee9f (127.0.98.1:44223), 56441041aae649e390d65ec16ad9c1bd (127.0.98.2:35203)
W20260620 09:09:19.463996 20055 tablet.cc:2369] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:19.468099 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.463903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44223 (local address 127.0.98.3:54109)
0620 09:09:19.464197 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.464213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.464298 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.464898 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.464901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.464921 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:19.465068 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.465076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.466126 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.466132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.466984 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.466993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.467690 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.467693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.467799 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.467823 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.467894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.467962 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":26}
I20260620 09:09:19.468132 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.464477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54109 (local address 127.0.98.1:44223)
0620 09:09:19.464644 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.464648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.464661 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:09:19.464712 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.464714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.464752 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.464880 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:19.465213 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.465997 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.467114 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.467568 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.467675 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.467746 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.467925 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.468004 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 09:09:19.468802 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.464092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57903 (local address 127.0.98.2:35203)
0620 09:09:19.464217 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.464222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.465070 (+ 848us) server_negotiation.cc:407] Connection header received
0620 09:09:19.465170 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.465174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.465225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.465313 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:19.465575 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.466084 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.467567 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.468131 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.468234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.468302 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.468607 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.468684 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:09:19.468755 19740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fffb7ab78143d590f0e4ac115e0468" candidate_uuid: "b5056118e5344b59b6d55ca4f2fc999a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b651930504c4bac4623d7074aee9f" is_pre_election: true
I20260620 09:09:19.469143 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.463914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35203 (local address 127.0.98.3:57903)
0620 09:09:19.464951 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.464965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:19.465084 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:19.465323 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:19.465326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:19.465349 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:19.465465 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.465471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.466210 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.466213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.467422 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:19.467434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.468378 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.468381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:19.468477 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.468498 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 09:09:19.468925 (+ 427us) client_negotiation.cc:240] Negotiation successful
0620 09:09:19.468991 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":643,"thread_start_us":137,"threads_started":1}
I20260620 09:09:19.469173 19740 raft_consensus.cc:2454] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5056118e5344b59b6d55ca4f2fc999a in term 0.
I20260620 09:09:19.469759 19943 leader_election.cc:302] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5056118e5344b59b6d55ca4f2fc999a, ce7b651930504c4bac4623d7074aee9f; no voters:
I20260620 09:09:19.469899 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fffb7ab78143d590f0e4ac115e0468" candidate_uuid: "b5056118e5344b59b6d55ca4f2fc999a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56441041aae649e390d65ec16ad9c1bd" is_pre_election: true
I20260620 09:09:19.470074 20078 raft_consensus.cc:2781] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:19.470170 20078 raft_consensus.cc:482] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:19.470224 20078 raft_consensus.cc:3037] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:19.470202 19874 raft_consensus.cc:2454] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5056118e5344b59b6d55ca4f2fc999a in term 0.
I20260620 09:09:19.471395 20078 raft_consensus.cc:504] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.471768 20078 leader_election.cc:288] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [CANDIDATE]: Term 1 election: Requested vote from peers ce7b651930504c4bac4623d7074aee9f (127.0.98.1:44223), 56441041aae649e390d65ec16ad9c1bd (127.0.98.2:35203)
I20260620 09:09:19.472124 19740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fffb7ab78143d590f0e4ac115e0468" candidate_uuid: "b5056118e5344b59b6d55ca4f2fc999a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7b651930504c4bac4623d7074aee9f"
I20260620 09:09:19.472255 19740 raft_consensus.cc:3037] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:19.473188 19874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fffb7ab78143d590f0e4ac115e0468" candidate_uuid: "b5056118e5344b59b6d55ca4f2fc999a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56441041aae649e390d65ec16ad9c1bd"
I20260620 09:09:19.473315 19874 raft_consensus.cc:3037] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:19.474354 19874 raft_consensus.cc:2454] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5056118e5344b59b6d55ca4f2fc999a in term 1.
I20260620 09:09:19.474676 19944 leader_election.cc:302] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56441041aae649e390d65ec16ad9c1bd, b5056118e5344b59b6d55ca4f2fc999a; no voters:
I20260620 09:09:19.474874 20078 raft_consensus.cc:2781] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:19.474982 19740 raft_consensus.cc:2454] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5056118e5344b59b6d55ca4f2fc999a in term 1.
I20260620 09:09:19.475128 20078 raft_consensus.cc:686] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 1 LEADER]: Becoming Leader. State: Replica: b5056118e5344b59b6d55ca4f2fc999a, State: Running, Role: LEADER
I20260620 09:09:19.475467 20078 consensus_queue.cc:237] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } }
I20260620 09:09:19.478335 19593 catalog_manager.cc:5495] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a reported cstate change: term changed from 0 to 1, leader changed from <none> to b5056118e5344b59b6d55ca4f2fc999a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b5056118e5344b59b6d55ca4f2fc999a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5056118e5344b59b6d55ca4f2fc999a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33767 } health_report { overall_health: HEALTHY } } }
W20260620 09:09:19.486883 19921 tablet.cc:2369] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:09:19.512197 19787 tablet.cc:2369] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:19.517338 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.513700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.3:33767)
0620 09:09:19.513835 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.513839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.513852 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:09:19.513932 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.513934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.514026 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.514125 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:19.514355 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.514903 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.515776 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.516022 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.516106 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.516370 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:19.516836 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:19.516931 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.517000 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.517206 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:09:19.524040 19740 raft_consensus.cc:1261] T 94fffb7ab78143d590f0e4ac115e0468 P ce7b651930504c4bac4623d7074aee9f [term 1 FOLLOWER]: Refusing update from remote peer b5056118e5344b59b6d55ca4f2fc999a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:19.524040 19874 raft_consensus.cc:1261] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 1 FOLLOWER]: Refusing update from remote peer b5056118e5344b59b6d55ca4f2fc999a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:19.524940 20078 consensus_queue.cc:1035] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7b651930504c4bac4623d7074aee9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:19.525187 20081 consensus_queue.cc:1035] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56441041aae649e390d65ec16ad9c1bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:19.531286 20084 mvcc.cc:205] Tried to move back new op lower bound from 7298853107802664960 to 7298853107614044160. Current Snapshot: MvccSnapshot[applied={T|T < 7298853107802664960}]
I20260620 09:09:19.532243 20085 mvcc.cc:205] Tried to move back new op lower bound from 7298853107802664960 to 7298853107614044160. Current Snapshot: MvccSnapshot[applied={T|T < 7298853107802664960}]
I20260620 09:09:19.533720 20086 mvcc.cc:205] Tried to move back new op lower bound from 7298853107802664960 to 7298853107614044160. Current Snapshot: MvccSnapshot[applied={T|T < 7298853107802664960}]
I20260620 09:09:19.543354 19595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:19.543551 19595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:19.546283 19595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c21cd74163a47ff8977f7c80fd52e4e: Sending DeleteTablet for 3 replicas of tablet 94fffb7ab78143d590f0e4ac115e0468
I20260620 09:09:19.547113 19854 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46854
I20260620 09:09:19.547120 19720 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45694
I20260620 09:09:19.547158 19988 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fffb7ab78143d590f0e4ac115e0468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39104
I20260620 09:09:19.547653 20094 tablet_replica.cc:330] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: stopping tablet replica
I20260620 09:09:19.547982 20094 raft_consensus.cc:2229] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:19.548321 20094 raft_consensus.cc:2258] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:19.548368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19657
I20260620 09:09:19.548990 20096 tablet_replica.cc:330] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: stopping tablet replica
I20260620 09:09:19.549319 20096 raft_consensus.cc:2229] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:19.549666 20096 raft_consensus.cc:2258] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:19.550413 20094 ts_tablet_manager.cc:1905] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:19.550907 20096 ts_tablet_manager.cc:1905] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:19.553259 20096 ts_tablet_manager.cc:1918] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:19.553428 20096 log.cc:1196] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/94fffb7ab78143d590f0e4ac115e0468
I20260620 09:09:19.553794 20094 ts_tablet_manager.cc:1918] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:19.553898 20094 log.cc:1196] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/94fffb7ab78143d590f0e4ac115e0468
I20260620 09:09:19.553855 20096 ts_tablet_manager.cc:1939] T 94fffb7ab78143d590f0e4ac115e0468 P b5056118e5344b59b6d55ca4f2fc999a: Deleting consensus metadata
I20260620 09:09:19.554217 20094 ts_tablet_manager.cc:1939] T 94fffb7ab78143d590f0e4ac115e0468 P 56441041aae649e390d65ec16ad9c1bd: Deleting consensus metadata
I20260620 09:09:19.555313 19579 catalog_manager.cc:4841] TS b5056118e5344b59b6d55ca4f2fc999a (127.0.98.3:33767): tablet 94fffb7ab78143d590f0e4ac115e0468 (table test-table [id=7d1f4bef821344f5a21065717dda3341]) successfully deleted
I20260620 09:09:19.555460 19582 catalog_manager.cc:4841] TS 56441041aae649e390d65ec16ad9c1bd (127.0.98.2:35203): tablet 94fffb7ab78143d590f0e4ac115e0468 (table test-table [id=7d1f4bef821344f5a21065717dda3341]) successfully deleted
W20260620 09:09:19.557451 19581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44223 (error 108)
I20260620 09:09:19.558426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19791
I20260620 09:09:19.560174 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.559743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44223 (local address 127.0.0.1:45710)
0620 09:09:19.559986 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:19.560064 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260620 09:09:19.560348 19581 catalog_manager.cc:4584] TS ce7b651930504c4bac4623d7074aee9f (127.0.98.1:44223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94fffb7ab78143d590f0e4ac115e0468: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44223: connect: Connection refused (error 111)
I20260620 09:09:19.564754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19925
I20260620 09:09:19.569859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 19563
2026-06-20T09:09:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2458 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T09:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:19Z Disabled control of system clock
I20260620 09:09:19.600557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:19.714385 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:19.714679 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:19.714717 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:19.718859 20101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:19.718946 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:19.718962 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:19.718981 20101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:19.718997 20101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:19.723976 20101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:19.725196 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:19.726707 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:19.735626 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:19.735673 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:19.735659 20101 server_base.cc:1034] running on GCE node
W20260620 09:09:19.735761 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:19.736584 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:19.737802 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:19.738971 20101 hybrid_clock.cc:648] HybridClock initialized: now 1781946559738953 us; error 33 us; skew 500 ppm
I20260620 09:09:19.740944 20101 webserver.cc:460] Webserver started at http://127.0.98.62:44305/ using document root <none> and password file <none>
I20260620 09:09:19.741602 20101 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:19.741668 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:19.741916 20101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:19.744092 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "02d55ab73b9642ee95b0056d26c44304"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "430ec8afb0a4ed09e1266e00da69fb9b"
server_key_iv: "21ad39d0296a2872f82cf91b5fb02231"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.744700 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "02d55ab73b9642ee95b0056d26c44304"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "430ec8afb0a4ed09e1266e00da69fb9b"
server_key_iv: "21ad39d0296a2872f82cf91b5fb02231"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.748704 20101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 09:09:19.751322 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:19.752641 20101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:19.752794 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "02d55ab73b9642ee95b0056d26c44304"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "430ec8afb0a4ed09e1266e00da69fb9b"
server_key_iv: "21ad39d0296a2872f82cf91b5fb02231"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.752918 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:19.765935 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:19.766701 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:19.766914 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:19.774242 20101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40399
I20260620 09:09:19.774263 20167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40399 every 8 connection(s)
I20260620 09:09:19.775238 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:19.776752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20101
I20260620 09:09:19.776901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:19.777277 392 external_mini_cluster.cc:1428] Setting key 6924e2859a8ec723cb0c442af043d1b1
I20260620 09:09:19.778826 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:19.784574 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: Bootstrap starting.
I20260620 09:09:19.786580 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:19.778694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.62:40399)
0620 09:09:19.779336 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:19.779350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:19.779385 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:09:19.780040 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:19.780056 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:19.780447 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:19.780784 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:19.781241 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.782247 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.783140 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:19.783615 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:19.783785 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:19.784738 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:19.784745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:19.784763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:19.785250 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:19.785308 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:19.785329 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:19.785442 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:19.785600 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 09:09:19.785858 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260620 09:09:19.787077 20168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:19.787835 20168 log.cc:824] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:19.789974 20168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: No bootstrap required, opened a new log
I20260620 09:09:19.793756 20168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260620 09:09:19.794016 20168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:19.794102 20168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d55ab73b9642ee95b0056d26c44304, State: Initialized, Role: FOLLOWER
I20260620 09:09:19.794574 20168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260620 09:09:19.794724 20168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:19.794811 20168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:19.794914 20168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:19.795950 20168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260620 09:09:19.796291 20168 leader_election.cc:302] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02d55ab73b9642ee95b0056d26c44304; no voters:
I20260620 09:09:19.796599 20168 leader_election.cc:288] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:19.796736 20173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:19.796983 20173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [term 1 LEADER]: Becoming Leader. State: Replica: 02d55ab73b9642ee95b0056d26c44304, State: Running, Role: LEADER
I20260620 09:09:19.797297 20173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } }
I20260620 09:09:19.797911 20168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:19.799046 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02d55ab73b9642ee95b0056d26c44304. Latest consensus state: current_term: 1 leader_uuid: "02d55ab73b9642ee95b0056d26c44304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
I20260620 09:09:19.799058 20174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02d55ab73b9642ee95b0056d26c44304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d55ab73b9642ee95b0056d26c44304" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40399 } } }
I20260620 09:09:19.799192 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:19.799192 20174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:19.799830 20182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:19.802783 20182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:19.809201 20182 catalog_manager.cc:1347] Generated new cluster ID: 7a01b301146741b7a01798f8cb72aa57
I20260620 09:09:19.809271 20182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:19.829027 20182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:19.830027 20182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:19.836752 20182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: Generated new TSK 0
I20260620 09:09:19.837652 20182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:19.841228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 09:09:19.957716 20192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:19.958025 20192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:19.958068 20192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:19.962276 20192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:19.962343 20192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:19.962405 20192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 09:09:19.967382 20192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:19.968550 20192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:19.970080 20192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:19.980108 20197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:19.980171 20198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:19.980166 20192 server_base.cc:1034] running on GCE node
W20260620 09:09:19.980103 20200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:19.980707 20192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:19.981437 20192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:19.982626 20192 hybrid_clock.cc:648] HybridClock initialized: now 1781946559982597 us; error 50 us; skew 500 ppm
I20260620 09:09:19.984540 20192 webserver.cc:460] Webserver started at http://127.0.98.1:41867/ using document root <none> and password file <none>
I20260620 09:09:19.985208 20192 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:19.985301 20192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:19.985538 20192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:19.987620 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data/instance:
uuid: "346acbc207384dfea62f99d8d88f3bf7"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "0c66e056a63f47d62532222b11591324"
server_key_iv: "b297545a4a1ccea280863a7d785fd259"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.988207 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal/instance:
uuid: "346acbc207384dfea62f99d8d88f3bf7"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "0c66e056a63f47d62532222b11591324"
server_key_iv: "b297545a4a1ccea280863a7d785fd259"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.991997 20192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 09:09:19.994575 20206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:19.995777 20192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:19.995962 20192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal
uuid: "346acbc207384dfea62f99d8d88f3bf7"
format_stamp: "Formatted at 2026-06-20 09:09:19 on dist-test-slave-92t4"
server_key: "0c66e056a63f47d62532222b11591324"
server_key_iv: "b297545a4a1ccea280863a7d785fd259"
server_key_version: "encryptionkey@0"
I20260620 09:09:19.996116 20192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:20.012975 20192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:20.013794 20192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:20.014021 20192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:20.014658 20192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:20.015661 20192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:20.015733 20192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.015805 20192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:20.015856 20192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.025104 20192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40093
I20260620 09:09:20.025162 20319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40093 every 8 connection(s)
I20260620 09:09:20.026086 20192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/data/info.pb
I20260620 09:09:20.028417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20192
I20260620 09:09:20.028558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-0/wal/instance
I20260620 09:09:20.028853 392 external_mini_cluster.cc:1428] Setting key 264cca7c8c156dfc0f1808013b73390e
I20260620 09:09:20.031090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:20.036052 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.027996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48517 (local address 127.0.98.62:40399)
0620 09:09:20.028143 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.028148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.029298 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 09:09:20.030912 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.030915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.030981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.031065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:20.032048 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.032592 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.033829 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.034079 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.034200 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.035475 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:20.035485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:20.035488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:20.035501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:20.035520 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:20.035528 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:20.035608 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.035781 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.035860 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:20.036931 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.028371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.1:48517)
0620 09:09:20.029136 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.029190 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.030693 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.031272 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.031282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.031390 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:20.031903 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.031916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.032729 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.032737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.033703 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.033712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.034245 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.034249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.034429 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.035147 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:20.035149 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:20.035191 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:20.035259 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:20.035622 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:20.035633 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:20.035639 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.035780 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.036081 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260620 09:09:20.038383 20320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260620 09:09:20.038681 20320 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:20.039278 20320 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260620 09:09:20.040989 20132 ts_manager.cc:194] Registered new tserver with Master: 346acbc207384dfea62f99d8d88f3bf7 (127.0.98.1:40093)
I20260620 09:09:20.042680 20132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48517
W20260620 09:09:20.152981 20323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:20.153380 20323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:20.153465 20323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:20.157815 20323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:20.157904 20323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:20.157994 20323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 09:09:20.163098 20323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:20.164317 20323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:20.165880 20323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:20.175961 20323 server_base.cc:1034] running on GCE node
W20260620 09:09:20.175930 20331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:20.175974 20328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:20.176502 20329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:20.176939 20323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:20.177696 20323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:20.178891 20323 hybrid_clock.cc:648] HybridClock initialized: now 1781946560178843 us; error 125 us; skew 500 ppm
I20260620 09:09:20.186239 20323 webserver.cc:460] Webserver started at http://127.0.98.2:37389/ using document root <none> and password file <none>
I20260620 09:09:20.186896 20323 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:20.186967 20323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:20.187202 20323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:20.189518 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data/instance:
uuid: "61544b879c254f9c89e37929f1fd5bed"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "db3f0b3c8bfebc14a01c4d54a1074ce5"
server_key_iv: "93a1392fcefbfd2128d14ba992658c21"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.190137 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61544b879c254f9c89e37929f1fd5bed"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "db3f0b3c8bfebc14a01c4d54a1074ce5"
server_key_iv: "93a1392fcefbfd2128d14ba992658c21"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.194072 20323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:09:20.196637 20337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.197969 20323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:20.198108 20323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal
uuid: "61544b879c254f9c89e37929f1fd5bed"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "db3f0b3c8bfebc14a01c4d54a1074ce5"
server_key_iv: "93a1392fcefbfd2128d14ba992658c21"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.198256 20323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:20.219453 20323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:20.220248 20323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:20.220468 20323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:20.221164 20323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:20.222172 20323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:20.222220 20323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.222292 20323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:20.222323 20323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.231817 20323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33421
I20260620 09:09:20.231838 20450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33421 every 8 connection(s)
I20260620 09:09:20.232800 20323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/data/info.pb
I20260620 09:09:20.239166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20323
I20260620 09:09:20.239282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal/instance
I20260620 09:09:20.239598 392 external_mini_cluster.cc:1428] Setting key f1152116a1d4963e8a36677e8b2d66cf
I20260620 09:09:20.241936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 09:09:20.242913 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.234907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42271 (local address 127.0.98.62:40399)
0620 09:09:20.235054 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.235058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.236202 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 09:09:20.237635 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.237637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.237690 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.237768 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:20.238699 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.239316 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.240702 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.240989 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.241148 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.242450 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:20.242453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:20.242456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:20.242468 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:20.242487 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:20.242494 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:20.242570 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.242742 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.242798 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 09:09:20.244077 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.235354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.2:42271)
0620 09:09:20.236026 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.236092 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.237426 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.237917 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.237924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.238016 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:20.238560 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.238574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.239477 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.239480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.240579 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.240591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.241199 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.241203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.241407 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.242108 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:20.242123 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:20.242195 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:20.242288 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:20.242583 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:20.242594 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:20.242603 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.242739 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.243080 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260620 09:09:20.245357 20451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260620 09:09:20.245617 20451 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:20.246210 20451 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260620 09:09:20.247313 20132 ts_manager.cc:194] Registered new tserver with Master: 61544b879c254f9c89e37929f1fd5bed (127.0.98.2:33421)
I20260620 09:09:20.248219 20132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42271
W20260620 09:09:20.359390 20454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:20.359714 20454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:20.359754 20454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:20.364054 20454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:20.364136 20454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:20.364197 20454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 09:09:20.369321 20454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:20.370465 20454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:20.371903 20454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:20.382218 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:20.382216 20459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:20.382346 20454 server_base.cc:1034] running on GCE node
W20260620 09:09:20.382216 20460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:20.382905 20454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:20.383530 20454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:20.384708 20454 hybrid_clock.cc:648] HybridClock initialized: now 1781946560384692 us; error 29 us; skew 500 ppm
I20260620 09:09:20.387099 20454 webserver.cc:460] Webserver started at http://127.0.98.3:35571/ using document root <none> and password file <none>
I20260620 09:09:20.387750 20454 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:20.387850 20454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:20.388077 20454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:20.390455 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data/instance:
uuid: "eec6b676adb54589a4728131b88f8106"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "b8e49263b4cc328c2e3f3b3051e3f6be"
server_key_iv: "1e92b37ea226524deeb34360ec864c9e"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.391075 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eec6b676adb54589a4728131b88f8106"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "b8e49263b4cc328c2e3f3b3051e3f6be"
server_key_iv: "1e92b37ea226524deeb34360ec864c9e"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.395198 20454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:20.398072 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.399709 20454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 09:09:20.399832 20454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal
uuid: "eec6b676adb54589a4728131b88f8106"
format_stamp: "Formatted at 2026-06-20 09:09:20 on dist-test-slave-92t4"
server_key: "b8e49263b4cc328c2e3f3b3051e3f6be"
server_key_iv: "1e92b37ea226524deeb34360ec864c9e"
server_key_version: "encryptionkey@0"
I20260620 09:09:20.400025 20454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:20.415768 20454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:20.416496 20454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:20.416654 20454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:20.417302 20454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 09:09:20.418656 20454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 09:09:20.418708 20454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.418751 20454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 09:09:20.418767 20454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:20.427383 20454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46161
I20260620 09:09:20.427464 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46161 every 8 connection(s)
I20260620 09:09:20.428397 20454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/data/info.pb
I20260620 09:09:20.429198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20454
I20260620 09:09:20.429313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal/instance
I20260620 09:09:20.429632 392 external_mini_cluster.cc:1428] Setting key 92ceb8499ee618a60415111a7bc9dc94
I20260620 09:09:20.438292 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.430463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36185 (local address 127.0.98.62:40399)
0620 09:09:20.431957 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.431962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.431978 (+ 16us) server_negotiation.cc:407] Connection header received
0620 09:09:20.433435 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.433439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.433495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.433583 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:20.434527 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.435073 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.436253 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.436489 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.436613 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.437759 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:20.437763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:20.437766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:20.437780 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:20.437808 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:20.437815 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:20.437893 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.438068 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.438137 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410}
I20260620 09:09:20.439234 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.430962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40399 (local address 127.0.98.3:36185)
0620 09:09:20.431722 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.431780 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.433217 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.433771 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.433778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.433870 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:20.434383 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.434395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.435211 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.435214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.436121 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.436130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.436657 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.436660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.436832 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.437483 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 09:09:20.437484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:20.437527 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 09:09:20.437597 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:20.437905 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:20.437917 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:20.437925 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.438083 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.438407 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260620 09:09:20.440538 20582 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40399
I20260620 09:09:20.440811 20582 heartbeater.cc:464] Registering TS with master...
I20260620 09:09:20.441455 20582 heartbeater.cc:511] Master 127.0.98.62:40399 requested a full tablet report, sending...
I20260620 09:09:20.442706 20132 ts_manager.cc:194] Registered new tserver with Master: eec6b676adb54589a4728131b88f8106 (127.0.98.3:46161)
I20260620 09:09:20.443523 20132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36185
I20260620 09:09:20.443820 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 09:09:20.449954 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.445926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.98.62:40399)
0620 09:09:20.446059 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.446064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.446256 (+ 192us) server_negotiation.cc:407] Connection header received
0620 09:09:20.446414 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.446417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.446464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.446556 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:20.446824 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.447331 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.448235 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.448466 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.448605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.449482 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:20.449486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:20.449489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:20.449502 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:20.449531 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:20.449539 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:20.449629 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.449752 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.449809 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:09:20.452889 20132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 09:09:20.455947 20132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 09:09:20.471438 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.464949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.98.3:46161)
0620 09:09:20.466198 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.466206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.466225 (+ 19us) server_negotiation.cc:407] Connection header received
0620 09:09:20.466312 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.466319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.466569 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.466732 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:20.467035 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468183 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.469901 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.470607 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.470703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.470894 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.471153 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.471261 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":134,"threads_started":1}
I20260620 09:09:20.471446 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.464835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33421 (local address 127.0.0.1:48770)
0620 09:09:20.465817 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.465832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.465928 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.466912 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.466916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.466935 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:20.467162 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.467170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468482 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.468489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.469622 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.469631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.470720 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.470724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.470913 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.470984 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.471104 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.471267 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":223,"thread_start_us":99,"threads_started":1}
I20260620 09:09:20.471969 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.465324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.2:33421)
0620 09:09:20.466168 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.466181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.466204 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:09:20.466282 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.466289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.466606 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.466759 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:20.467297 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468379 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.469747 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.470580 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.470700 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.470906 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.471655 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.471762 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":123,"threads_started":1}
I20260620 09:09:20.472579 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.465549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.0.1:51332)
0620 09:09:20.465893 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.465907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.466000 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.466955 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.466958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.466980 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:20.467194 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.467203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468174 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.468179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.469784 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.469789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.472137 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.472141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.472231 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.472260 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.472353 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.472434 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":185,"spinlock_wait_cycles":1920,"thread_start_us":135,"threads_started":1}
I20260620 09:09:20.472651 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.465799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.98.1:40093)
0620 09:09:20.466435 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.466446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.466466 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:20.466536 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.466542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.466711 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.466829 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:20.467326 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468057 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.470960 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.471766 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.471876 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.472070 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.472387 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.472486 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":127,"threads_started":1}
I20260620 09:09:20.471446 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.464772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46161 (local address 127.0.0.1:38600)
0620 09:09:20.465228 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.465283 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.465414 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.466723 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.466728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.466750 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:20.466919 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.466931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.468290 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.468293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.469778 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.469790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.470720 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.470724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.470908 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.470984 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.471104 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.471260 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":196,"thread_start_us":93,"threads_started":1}
I20260620 09:09:20.475082 20385 tablet_service.cc:1467] Processing CreateTablet for tablet f69becb3713d45098354595ba504acbb (DEFAULT_TABLE table=test-table [id=5b3e6b7461f04e5d9505d9dd45342c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:20.475692 20516 tablet_service.cc:1467] Processing CreateTablet for tablet f69becb3713d45098354595ba504acbb (DEFAULT_TABLE table=test-table [id=5b3e6b7461f04e5d9505d9dd45342c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:20.475538 20254 tablet_service.cc:1467] Processing CreateTablet for tablet f69becb3713d45098354595ba504acbb (DEFAULT_TABLE table=test-table [id=5b3e6b7461f04e5d9505d9dd45342c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 09:09:20.476181 20385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69becb3713d45098354595ba504acbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:20.476632 20516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69becb3713d45098354595ba504acbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:20.476632 20254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f69becb3713d45098354595ba504acbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:20.482959 20599 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Bootstrap starting.
I20260620 09:09:20.484475 20600 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Bootstrap starting.
I20260620 09:09:20.485505 20601 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Bootstrap starting.
I20260620 09:09:20.485683 20599 tablet_bootstrap.cc:654] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:20.486581 20599 log.cc:824] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:20.486768 20600 tablet_bootstrap.cc:654] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:20.487442 20601 tablet_bootstrap.cc:654] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:20.487555 20600 log.cc:824] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:20.488502 20601 log.cc:824] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:20.488852 20599 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: No bootstrap required, opened a new log
I20260620 09:09:20.489090 20599 ts_tablet_manager.cc:1397] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:20.490417 20601 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: No bootstrap required, opened a new log
I20260620 09:09:20.490633 20601 ts_tablet_manager.cc:1397] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 09:09:20.490820 20600 tablet_bootstrap.cc:492] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: No bootstrap required, opened a new log
I20260620 09:09:20.491009 20600 ts_tablet_manager.cc:1397] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 09:09:20.493603 20599 raft_consensus.cc:348] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.493887 20599 raft_consensus.cc:374] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:20.493943 20599 raft_consensus.cc:729] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61544b879c254f9c89e37929f1fd5bed, State: Initialized, Role: FOLLOWER
I20260620 09:09:20.494022 20601 raft_consensus.cc:348] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.494297 20601 raft_consensus.cc:374] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:20.494361 20601 raft_consensus.cc:729] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346acbc207384dfea62f99d8d88f3bf7, State: Initialized, Role: FOLLOWER
I20260620 09:09:20.494439 20599 consensus_queue.cc:260] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.495133 20451 heartbeater.cc:503] Master 127.0.98.62:40399 was elected leader, sending a full tablet report...
I20260620 09:09:20.495483 20599 ts_tablet_manager.cc:1428] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:20.495373 20600 raft_consensus.cc:348] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.495594 20600 raft_consensus.cc:374] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:20.495637 20600 raft_consensus.cc:729] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec6b676adb54589a4728131b88f8106, State: Initialized, Role: FOLLOWER
I20260620 09:09:20.494802 20601 consensus_queue.cc:260] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.496037 20600 consensus_queue.cc:260] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.496533 20601 ts_tablet_manager.cc:1428] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 09:09:20.496608 20320 heartbeater.cc:503] Master 127.0.98.62:40399 was elected leader, sending a full tablet report...
I20260620 09:09:20.496837 20600 ts_tablet_manager.cc:1428] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 09:09:20.497267 20582 heartbeater.cc:503] Master 127.0.98.62:40399 was elected leader, sending a full tablet report...
W20260620 09:09:20.527480 20321 tablet.cc:2369] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 09:09:20.679550 20583 tablet.cc:2369] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:20.700104 20605 raft_consensus.cc:482] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 09:09:20.700376 20605 raft_consensus.cc:504] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.701819 20605 leader_election.cc:288] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eec6b676adb54589a4728131b88f8106 (127.0.98.3:46161), 346acbc207384dfea62f99d8d88f3bf7 (127.0.98.1:40093)
I20260620 09:09:20.706218 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.702427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38901 (local address 127.0.98.1:40093)
0620 09:09:20.702538 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.702542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.702759 (+ 217us) server_negotiation.cc:407] Connection header received
0620 09:09:20.702867 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.702870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.702955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.703044 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:20.703394 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.703975 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705193 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.705633 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705714 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.705766 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.706000 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.706086 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 09:09:20.706233 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.702323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.98.2:38901)
0620 09:09:20.702678 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.702690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.702768 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.703067 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.703069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.703084 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:20.703230 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.703237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.704093 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.704095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.705020 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.705030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.705759 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.705872 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.705896 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.705978 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.706050 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":41,"thread_start_us":118,"threads_started":1}
I20260620 09:09:20.706331 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.702012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46161 (local address 127.0.98.2:57531)
0620 09:09:20.702416 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.702434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:20.702530 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:20.702915 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:20.702918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:20.702936 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 09:09:20.703086 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.703095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.704005 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.704011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.705178 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:20.705189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.705895 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:20.705999 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.706024 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 09:09:20.706099 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 09:09:20.706167 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":83}
I20260620 09:09:20.706337 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.702127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57531 (local address 127.0.98.3:46161)
0620 09:09:20.702305 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.702310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.702630 (+ 320us) server_negotiation.cc:407] Connection header received
0620 09:09:20.702696 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.702701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.702757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.702832 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 09:09:20.703221 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.703897 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705298 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.705751 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.705857 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.705918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.706137 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.706210 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 09:09:20.706923 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69becb3713d45098354595ba504acbb" candidate_uuid: "61544b879c254f9c89e37929f1fd5bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346acbc207384dfea62f99d8d88f3bf7" is_pre_election: true
I20260620 09:09:20.706926 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69becb3713d45098354595ba504acbb" candidate_uuid: "61544b879c254f9c89e37929f1fd5bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec6b676adb54589a4728131b88f8106" is_pre_election: true
I20260620 09:09:20.707244 20274 raft_consensus.cc:2454] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61544b879c254f9c89e37929f1fd5bed in term 0.
I20260620 09:09:20.707252 20536 raft_consensus.cc:2454] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61544b879c254f9c89e37929f1fd5bed in term 0.
I20260620 09:09:20.707736 20341 leader_election.cc:302] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61544b879c254f9c89e37929f1fd5bed, eec6b676adb54589a4728131b88f8106; no voters:
I20260620 09:09:20.708029 20605 raft_consensus.cc:2781] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 09:09:20.708117 20605 raft_consensus.cc:482] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 09:09:20.708163 20605 raft_consensus.cc:3037] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:20.709228 20605 raft_consensus.cc:504] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.709590 20605 leader_election.cc:288] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [CANDIDATE]: Term 1 election: Requested vote from peers eec6b676adb54589a4728131b88f8106 (127.0.98.3:46161), 346acbc207384dfea62f99d8d88f3bf7 (127.0.98.1:40093)
I20260620 09:09:20.709955 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69becb3713d45098354595ba504acbb" candidate_uuid: "61544b879c254f9c89e37929f1fd5bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eec6b676adb54589a4728131b88f8106"
I20260620 09:09:20.710006 20274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f69becb3713d45098354595ba504acbb" candidate_uuid: "61544b879c254f9c89e37929f1fd5bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346acbc207384dfea62f99d8d88f3bf7"
I20260620 09:09:20.710075 20536 raft_consensus.cc:3037] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:20.710199 20274 raft_consensus.cc:3037] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:20.711179 20536 raft_consensus.cc:2454] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61544b879c254f9c89e37929f1fd5bed in term 1.
I20260620 09:09:20.711272 20274 raft_consensus.cc:2454] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61544b879c254f9c89e37929f1fd5bed in term 1.
I20260620 09:09:20.711539 20341 leader_election.cc:302] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61544b879c254f9c89e37929f1fd5bed, eec6b676adb54589a4728131b88f8106; no voters:
I20260620 09:09:20.711714 20605 raft_consensus.cc:2781] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:20.711948 20605 raft_consensus.cc:686] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 1 LEADER]: Becoming Leader. State: Replica: 61544b879c254f9c89e37929f1fd5bed, State: Running, Role: LEADER
I20260620 09:09:20.712355 20605 consensus_queue.cc:237] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } }
I20260620 09:09:20.716130 20132 catalog_manager.cc:5495] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 61544b879c254f9c89e37929f1fd5bed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "61544b879c254f9c89e37929f1fd5bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61544b879c254f9c89e37929f1fd5bed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 } health_report { overall_health: UNKNOWN } } }
W20260620 09:09:20.734062 20452 tablet.cc:2369] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 09:09:20.766325 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:20.762761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.0.98.2:33421)
0620 09:09:20.762901 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:20.762905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:20.762941 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:09:20.763020 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:20.763022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:20.763082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:20.763167 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:20.763400 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.763950 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.764786 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:20.765120 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:20.765205 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:20.765477 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:20.765976 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:20.766072 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:20.766133 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 09:09:20.766187 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 09:09:20.772663 20274 raft_consensus.cc:1261] T f69becb3713d45098354595ba504acbb P 346acbc207384dfea62f99d8d88f3bf7 [term 1 FOLLOWER]: Refusing update from remote peer 61544b879c254f9c89e37929f1fd5bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:20.772714 20536 raft_consensus.cc:1261] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 1 FOLLOWER]: Refusing update from remote peer 61544b879c254f9c89e37929f1fd5bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:20.773456 20605 consensus_queue.cc:1035] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "346acbc207384dfea62f99d8d88f3bf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:20.773706 20609 consensus_queue.cc:1035] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "eec6b676adb54589a4728131b88f8106" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:20.782153 20612 mvcc.cc:205] Tried to move back new op lower bound from 7298853112917065728 to 7298853112682123264. Current Snapshot: MvccSnapshot[applied={T|T < 7298853112917065728}]
I20260620 09:09:20.787065 20614 mvcc.cc:205] Tried to move back new op lower bound from 7298853112917065728 to 7298853112682123264. Current Snapshot: MvccSnapshot[applied={T|T < 7298853112917065728 or (T in {7298853112917065728})}]
I20260620 09:09:20.793371 20131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:20.793548 20131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 09:09:20.796083 20131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02d55ab73b9642ee95b0056d26c44304: Sending DeleteTablet for 3 replicas of tablet f69becb3713d45098354595ba504acbb
I20260620 09:09:20.796970 20385 tablet_service.cc:1514] Processing DeleteTablet for tablet f69becb3713d45098354595ba504acbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:20 UTC) from {username='slave'} at 127.0.0.1:48770
I20260620 09:09:20.797101 20516 tablet_service.cc:1514] Processing DeleteTablet for tablet f69becb3713d45098354595ba504acbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:20 UTC) from {username='slave'} at 127.0.0.1:38600
I20260620 09:09:20.797535 20254 tablet_service.cc:1514] Processing DeleteTablet for tablet f69becb3713d45098354595ba504acbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 09:09:20 UTC) from {username='slave'} at 127.0.0.1:51332
I20260620 09:09:20.797916 20622 tablet_replica.cc:330] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: stopping tablet replica
I20260620 09:09:20.798171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20192
I20260620 09:09:20.798990 20622 raft_consensus.cc:2229] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 1 LEADER]: Raft consensus shutting down.
I20260620 09:09:20.799394 20622 raft_consensus.cc:2258] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:20.799870 20623 tablet_replica.cc:330] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: stopping tablet replica
I20260620 09:09:20.800220 20623 raft_consensus.cc:2229] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 09:09:20.800560 20623 raft_consensus.cc:2258] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 09:09:20.801131 20622 ts_tablet_manager.cc:1905] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:20.802145 20623 ts_tablet_manager.cc:1905] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 09:09:20.804839 20622 ts_tablet_manager.cc:1918] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:20.804975 20622 log.cc:1196] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-1/wal/wals/f69becb3713d45098354595ba504acbb
I20260620 09:09:20.805140 20623 ts_tablet_manager.cc:1918] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 09:09:20.805236 20623 log.cc:1196] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Deleting WAL directory at /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781946309842113-392-0/minicluster-data/ts-2/wal/wals/f69becb3713d45098354595ba504acbb
I20260620 09:09:20.805387 20622 ts_tablet_manager.cc:1939] T f69becb3713d45098354595ba504acbb P 61544b879c254f9c89e37929f1fd5bed: Deleting consensus metadata
I20260620 09:09:20.805543 20623 ts_tablet_manager.cc:1939] T f69becb3713d45098354595ba504acbb P eec6b676adb54589a4728131b88f8106: Deleting consensus metadata
I20260620 09:09:20.806759 20119 catalog_manager.cc:4841] TS eec6b676adb54589a4728131b88f8106 (127.0.98.3:46161): tablet f69becb3713d45098354595ba504acbb (table test-table [id=5b3e6b7461f04e5d9505d9dd45342c46]) successfully deleted
W20260620 09:09:20.807947 20119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40093 (error 108)
I20260620 09:09:20.808596 20116 catalog_manager.cc:4841] TS 61544b879c254f9c89e37929f1fd5bed (127.0.98.2:33421): tablet f69becb3713d45098354595ba504acbb (table test-table [id=5b3e6b7461f04e5d9505d9dd45342c46]) successfully deleted
W20260620 09:09:20.809327 20593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:20.808942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40093 (local address 127.0.0.1:51344)
0620 09:09:20.809126 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:20.809208 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40093: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 09:09:20.809507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20323
W20260620 09:09:20.811115 20119 catalog_manager.cc:4584] TS 346acbc207384dfea62f99d8d88f3bf7 (127.0.98.1:40093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f69becb3713d45098354595ba504acbb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40093: connect: Connection reset by peer (error 104)
I20260620 09:09:20.815761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20454
I20260620 09:09:20.821590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20101
2026-06-20T09:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1250 ms)
[----------] 6 tests from EncryptionPolicyTest (10874 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:20 dist-test-slave-92t4 krb5kdc[20627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:20 dist-test-slave-92t4 krb5kdc[20627](info): set up 2 sockets
Jun 20 09:09:20 dist-test-slave-92t4 krb5kdc[20627](info): commencing operation
krb5kdc: starting...
W20260620 09:09:21.865315 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:21 dist-test-slave-92t4 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:21.993207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:22.111826 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:22.112128 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:22.112172 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:22.116513 20640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:22.116590 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:22.116609 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:22.116627 20640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:22.116669 20640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:22.122391 20640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:22.123629 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:22.124866 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:22.135615 20648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:22.135715 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:22.135808 20640 server_base.cc:1034] running on GCE node
W20260620 09:09:22.135615 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:22.137583 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:22.138792 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:22.140024 20640 hybrid_clock.cc:648] HybridClock initialized: now 1781946562139991 us; error 55 us; skew 500 ppm
Jun 20 09:09:22 dist-test-slave-92t4 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:22.143218 20640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:22.144270 20640 webserver.cc:460] Webserver started at http://127.0.0.1:41349/ using document root <none> and password file <none>
I20260620 09:09:22.144901 20640 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:22.145008 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:22.145283 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:22.147513 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "ec03c16e68a14dcb82000ea9a71d38ba"
format_stamp: "Formatted at 2026-06-20 09:09:22 on dist-test-slave-92t4"
server_key: "f925c506adc9965c6fa154536e9938f2"
server_key_iv: "6d151ba4fb9fad1141628ac35f64381a"
server_key_version: "encryptionkey@0"
I20260620 09:09:22.148128 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec03c16e68a14dcb82000ea9a71d38ba"
format_stamp: "Formatted at 2026-06-20 09:09:22 on dist-test-slave-92t4"
server_key: "f925c506adc9965c6fa154536e9938f2"
server_key_iv: "6d151ba4fb9fad1141628ac35f64381a"
server_key_version: "encryptionkey@0"
I20260620 09:09:22.152084 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 09:09:22.154706 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:22.156066 20640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:22.156210 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "ec03c16e68a14dcb82000ea9a71d38ba"
format_stamp: "Formatted at 2026-06-20 09:09:22 on dist-test-slave-92t4"
server_key: "f925c506adc9965c6fa154536e9938f2"
server_key_iv: "6d151ba4fb9fad1141628ac35f64381a"
server_key_version: "encryptionkey@0"
I20260620 09:09:22.156333 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:22.213476 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:22.216540 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:22.216758 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:22.223837 20640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36903
I20260620 09:09:22.223863 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36903 every 8 connection(s)
I20260620 09:09:22.224843 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:22.227633 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:22.231043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20640
I20260620 09:09:22.231197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:22.231513 392 external_mini_cluster.cc:1428] Setting key d30fef2c87e3bc76458b7e7944b312d8
I20260620 09:09:22.233976 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba: Bootstrap starting.
I20260620 09:09:22.236250 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:22.237017 20708 log.cc:824] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:22.239111 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba: No bootstrap required, opened a new log
Jun 20 09:09:22 dist-test-slave-92t4 krb5kdc[20627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:09:22.243885 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } }
I20260620 09:09:22.244182 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:22.244253 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec03c16e68a14dcb82000ea9a71d38ba, State: Initialized, Role: FOLLOWER
I20260620 09:09:22.244812 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } }
I20260620 09:09:22.244999 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:22.245105 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:22.245221 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:22.246395 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } }
I20260620 09:09:22.246762 20708 leader_election.cc:302] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec03c16e68a14dcb82000ea9a71d38ba; no voters:
I20260620 09:09:22.247123 20708 leader_election.cc:288] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:22.247200 20713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:22.247404 20713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [term 1 LEADER]: Becoming Leader. State: Replica: ec03c16e68a14dcb82000ea9a71d38ba, State: Running, Role: LEADER
I20260620 09:09:22.247692 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:22.233138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36357 (local address 127.0.0.1:36903)
0620 09:09:22.233596 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:22.233613 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:22.233652 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:09:22.234416 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:22.234437 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:22.234807 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:22.235176 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:22.236047 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:22.237220 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:22.238311 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:22.238823 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:22.239008 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:22.241946 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:22.241951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:22.241967 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:22.241996 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:22.244036 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:22.244678 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:22.244683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:22.244688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:22.244820 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:22.245303 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:22.245307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:22.245309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:22.245754 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:22.245959 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:22.246320 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 09:09:22.246617 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":139,"threads_started":1}
I20260620 09:09:22.247728 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } }
I20260620 09:09:22.248518 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:22.248987 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec03c16e68a14dcb82000ea9a71d38ba. Latest consensus state: current_term: 1 leader_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } } }
I20260620 09:09:22.249145 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:22.249413 20714 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03c16e68a14dcb82000ea9a71d38ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36903 } } }
I20260620 09:09:22.249522 20717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:22.249542 20714 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:22.251225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:22.252871 20717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:22.259429 20717 catalog_manager.cc:1347] Generated new cluster ID: 5d7e5a69b1d24ba1aece7c96446802ed
I20260620 09:09:22.259496 20717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:22.263072 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:22.253675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49653 (local address 127.0.0.1:36903)
0620 09:09:22.253794 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:22.253800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:22.254094 (+ 294us) server_negotiation.cc:407] Connection header received
0620 09:09:22.254210 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:22.254213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:22.254259 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:22.254340 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:22.255269 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:22.256017 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:22.257254 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:22.257557 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:22.257676 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:22.259362 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:22.259366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:22.259368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:22.259395 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:22.261187 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:22.261767 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:22.261770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:22.261772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:22.261855 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:22.262207 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:22.262210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:22.262211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:22.262401 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:22.262524 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:22.262816 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 09:09:22.262915 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 09:09:22.275676 20717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:22.277092 20717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:22.285729 20717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec03c16e68a14dcb82000ea9a71d38ba: Generated new TSK 0
I20260620 09:09:22.286631 20717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:22.292007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20640
W20260620 09:09:22.296847 20625 connection.cc:535] client connection to 127.0.0.1:36903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36903): Connection reset by peer (error 104)
2026-06-20T09:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:22 dist-test-slave-92t4 krb5kdc[20739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:22 dist-test-slave-92t4 krb5kdc[20739](info): set up 2 sockets
Jun 20 09:09:22 dist-test-slave-92t4 krb5kdc[20739](info): commencing operation
krb5kdc: starting...
W20260620 09:09:23.329454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:23 dist-test-slave-92t4 krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:23.456418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:23.570814 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:23.571174 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:23.571256 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:23.575452 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:23.575578 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:23.575619 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:23.575657 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:23.575690 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:23.581351 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:23.582633 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:23.583732 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:23.592670 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:23.592660 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:23.592639 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:23.593091 20752 server_base.cc:1034] running on GCE node
I20260620 09:09:23.593773 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:23.594993 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:23.596197 20752 hybrid_clock.cc:648] HybridClock initialized: now 1781946563596183 us; error 30 us; skew 500 ppm
Jun 20 09:09:23 dist-test-slave-92t4 krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:23.599102 20752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:23.600133 20752 webserver.cc:460] Webserver started at http://127.0.0.1:42069/ using document root <none> and password file <none>
I20260620 09:09:23.600726 20752 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:23.600787 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:23.600970 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:23.603327 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "00e10f2e093545c3a62f3cad01edd45c"
format_stamp: "Formatted at 2026-06-20 09:09:23 on dist-test-slave-92t4"
server_key: "c546950b0396b9237ef3b97963f2ba78"
server_key_iv: "4dfd3fedaf93fd82f11bd998aeeaeb63"
server_key_version: "encryptionkey@0"
I20260620 09:09:23.603899 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "00e10f2e093545c3a62f3cad01edd45c"
format_stamp: "Formatted at 2026-06-20 09:09:23 on dist-test-slave-92t4"
server_key: "c546950b0396b9237ef3b97963f2ba78"
server_key_iv: "4dfd3fedaf93fd82f11bd998aeeaeb63"
server_key_version: "encryptionkey@0"
I20260620 09:09:23.607940 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:23.610561 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:23.611948 20752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:23.612056 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "00e10f2e093545c3a62f3cad01edd45c"
format_stamp: "Formatted at 2026-06-20 09:09:23 on dist-test-slave-92t4"
server_key: "c546950b0396b9237ef3b97963f2ba78"
server_key_iv: "4dfd3fedaf93fd82f11bd998aeeaeb63"
server_key_version: "encryptionkey@0"
I20260620 09:09:23.612149 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:23.630017 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:23.633409 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:23.633584 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:23.641162 20752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39819
I20260620 09:09:23.641178 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39819 every 8 connection(s)
I20260620 09:09:23.642303 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:23.643332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20752
I20260620 09:09:23.643492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:23.643787 392 external_mini_cluster.cc:1428] Setting key ef6cbf2129bc930954d9935349d89052
I20260620 09:09:23.645773 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:23.651494 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c: Bootstrap starting.
I20260620 09:09:23.653280 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:23.645358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34135 (local address 127.0.0.1:39819)
0620 09:09:23.645818 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:23.645834 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:23.645873 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:09:23.646586 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:23.646603 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:23.646959 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:23.647333 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:23.647753 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:23.648768 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:23.649769 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:23.650258 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:23.650430 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:23.651424 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:23.651429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:23.651442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:23.651893 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:23.651950 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:23.651974 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:23.652128 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:23.652310 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 09:09:23.652461 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260620 09:09:23.654136 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:23.654917 20820 log.cc:824] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:23.656468 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:23.657153 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c: No bootstrap required, opened a new log
I20260620 09:09:23.661849 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } }
I20260620 09:09:23.662223 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:23.662298 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e10f2e093545c3a62f3cad01edd45c, State: Initialized, Role: FOLLOWER
I20260620 09:09:23.662829 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } }
I20260620 09:09:23.662992 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:23.658621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56265 (local address 127.0.0.1:39819)
0620 09:09:23.658748 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:23.658753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:23.658926 (+ 173us) server_negotiation.cc:407] Connection header received
0620 09:09:23.659012 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:23.659015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:23.659060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:23.659150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:23.659390 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:23.660234 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:23.661058 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:23.661376 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:23.661483 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:23.662423 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:23.662427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:23.662430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:23.662449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:23.662473 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:23.662484 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:23.662595 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:23.662782 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:09:23.662836 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 09:09:23.663022 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:23.663102 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:23.663204 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:23.664167 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } }
I20260620 09:09:23.664534 20820 leader_election.cc:302] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e10f2e093545c3a62f3cad01edd45c; no voters:
I20260620 09:09:23.664888 20820 leader_election.cc:288] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:23.665108 20830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:23.665343 20830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [term 1 LEADER]: Becoming Leader. State: Replica: 00e10f2e093545c3a62f3cad01edd45c, State: Running, Role: LEADER
I20260620 09:09:23.665647 20830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } }
I20260620 09:09:23.666340 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:23.667346 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e10f2e093545c3a62f3cad01edd45c. Latest consensus state: current_term: 1 leader_uuid: "00e10f2e093545c3a62f3cad01edd45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } } }
I20260620 09:09:23.667359 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e10f2e093545c3a62f3cad01edd45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e10f2e093545c3a62f3cad01edd45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39819 } } }
I20260620 09:09:23.667569 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:23.667851 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:23.668015 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:23.671330 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:23.677500 20839 catalog_manager.cc:1347] Generated new cluster ID: 488a6a4ba3c34463903a95444acf252b
I20260620 09:09:23.677587 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:23.691061 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:23.692077 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:23.719295 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e10f2e093545c3a62f3cad01edd45c: Generated new TSK 0
I20260620 09:09:23.720309 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:23.731891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20752
W20260620 09:09:23.737630 20737 connection.cc:535] client connection to 127.0.0.1:39819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39819): Connection reset by peer (error 104)
2026-06-20T09:09:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:23 dist-test-slave-92t4 krb5kdc[20851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:23 dist-test-slave-92t4 krb5kdc[20851](info): set up 2 sockets
Jun 20 09:09:23 dist-test-slave-92t4 krb5kdc[20851](info): commencing operation
krb5kdc: starting...
W20260620 09:09:24.773358 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:24 dist-test-slave-92t4 krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:24.898455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:25.018867 20864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:25.019152 20864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:25.019196 20864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:25.023351 20864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:25.023427 20864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:25.023447 20864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:25.023464 20864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:25.023516 20864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:25.029204 20864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:25.030395 20864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:25.031482 20864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:25.040462 20872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:25.040505 20869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:25.040565 20864 server_base.cc:1034] running on GCE node
W20260620 09:09:25.040500 20870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:25.041462 20864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:25.042709 20864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:25.043887 20864 hybrid_clock.cc:648] HybridClock initialized: now 1781946565043854 us; error 55 us; skew 500 ppm
Jun 20 09:09:25 dist-test-slave-92t4 krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:25.046986 20864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:25.048002 20864 webserver.cc:460] Webserver started at http://127.0.0.1:43095/ using document root <none> and password file <none>
I20260620 09:09:25.048610 20864 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:25.048704 20864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:25.048949 20864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:25.051159 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "e0971c55fcef4506b401b617651ec7de"
format_stamp: "Formatted at 2026-06-20 09:09:25 on dist-test-slave-92t4"
server_key: "b82d19d4addc4faac792c35a004c879a"
server_key_iv: "a674a3f5a93e2ddab6a5bc19975d2c70"
server_key_version: "encryptionkey@0"
I20260620 09:09:25.051769 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0971c55fcef4506b401b617651ec7de"
format_stamp: "Formatted at 2026-06-20 09:09:25 on dist-test-slave-92t4"
server_key: "b82d19d4addc4faac792c35a004c879a"
server_key_iv: "a674a3f5a93e2ddab6a5bc19975d2c70"
server_key_version: "encryptionkey@0"
I20260620 09:09:25.055899 20864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 09:09:25.058683 20879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:25.060068 20864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:25.060222 20864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "e0971c55fcef4506b401b617651ec7de"
format_stamp: "Formatted at 2026-06-20 09:09:25 on dist-test-slave-92t4"
server_key: "b82d19d4addc4faac792c35a004c879a"
server_key_iv: "a674a3f5a93e2ddab6a5bc19975d2c70"
server_key_version: "encryptionkey@0"
I20260620 09:09:25.060357 20864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:25.065608 20864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:25.068866 20864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:25.069162 20864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:25.077173 20864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260620 09:09:25.077186 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260620 09:09:25.078281 20864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:25.081497 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:25.085639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20864
I20260620 09:09:25.085836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:25.086131 392 external_mini_cluster.cc:1428] Setting key 920733fe87f66580edb8e9702a66adb0
I20260620 09:09:25.087949 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de: Bootstrap starting.
I20260620 09:09:25.090482 20932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:25.091355 20932 log.cc:824] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:25.091918 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:25.087671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53821 (local address 127.0.0.1:43687)
0620 09:09:25.088222 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:25.088239 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:25.088278 (+ 39us) server_negotiation.cc:407] Connection header received
0620 09:09:25.088950 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:25.088968 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:25.089351 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:25.089716 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:25.090565 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:25.090576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:25.090595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:25.090607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:25.090671 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:25.090692 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:25.090806 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:25.090960 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 09:09:25.091136 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":165,"threads_started":1}
I20260620 09:09:25.093451 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de: No bootstrap required, opened a new log
I20260620 09:09:25.094901 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:25.097594 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 09:09:25.097944 20932 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:25.098021 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0971c55fcef4506b401b617651ec7de, State: Initialized, Role: FOLLOWER
I20260620 09:09:25.098556 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 09:09:25.098727 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:25.098811 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:25.098834 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:25.097013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37961 (local address 127.0.0.1:43687)
0620 09:09:25.097172 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:25.097178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:25.097360 (+ 182us) server_negotiation.cc:407] Connection header received
0620 09:09:25.097503 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:25.097505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:25.097536 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:25.097602 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:25.098422 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:25.098426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:25.098428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:25.098440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:25.098471 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:25.098484 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:25.098576 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:25.098635 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 09:09:25.098676 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 09:09:25.098935 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:25.100059 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 09:09:25.100363 20932 leader_election.cc:302] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0971c55fcef4506b401b617651ec7de; no voters:
I20260620 09:09:25.100626 20932 leader_election.cc:288] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:25.100781 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:25.101020 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [term 1 LEADER]: Becoming Leader. State: Replica: e0971c55fcef4506b401b617651ec7de, State: Running, Role: LEADER
I20260620 09:09:25.101377 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 09:09:25.101940 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:25.103230 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0971c55fcef4506b401b617651ec7de. Latest consensus state: current_term: 1 leader_uuid: "e0971c55fcef4506b401b617651ec7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260620 09:09:25.103264 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0971c55fcef4506b401b617651ec7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0971c55fcef4506b401b617651ec7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260620 09:09:25.103353 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:25.103360 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:25.103782 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:25.107189 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:25.112915 20951 catalog_manager.cc:1347] Generated new cluster ID: 2bf26f144cc341eb913cf12364e1044f
I20260620 09:09:25.113018 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:25.143893 20951 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:25.144332 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:25.156103 20951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0971c55fcef4506b401b617651ec7de: Generated new TSK 0
I20260620 09:09:25.156915 20951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:25.165763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20864
2026-06-20T09:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:25 dist-test-slave-92t4 krb5kdc[20963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:25 dist-test-slave-92t4 krb5kdc[20963](info): set up 2 sockets
Jun 20 09:09:25 dist-test-slave-92t4 krb5kdc[20963](info): commencing operation
krb5kdc: starting...
W20260620 09:09:26.201714 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:26.323782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:26.439881 20976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:26.440181 20976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:26.440227 20976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:26.444408 20976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:26.444494 20976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:26.444514 20976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:26.444531 20976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:26.444597 20976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:26.450909 20976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:26.452088 20976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:26.453246 20976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:26.461769 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:26.461773 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:26.461800 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:26.462033 20976 server_base.cc:1034] running on GCE node
I20260620 09:09:26.462731 20976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:26.463919 20976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:26.465121 20976 hybrid_clock.cc:648] HybridClock initialized: now 1781946566465087 us; error 48 us; skew 500 ppm
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:26.467893 20976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:26.468895 20976 webserver.cc:460] Webserver started at http://127.0.0.1:40385/ using document root <none> and password file <none>
I20260620 09:09:26.469570 20976 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:26.469671 20976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:26.469967 20976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:26.472072 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5"
format_stamp: "Formatted at 2026-06-20 09:09:26 on dist-test-slave-92t4"
server_key: "682b18aeef495e213cec000472da40fd"
server_key_iv: "95de0637515afae9145058b1a2c7b513"
server_key_version: "encryptionkey@0"
I20260620 09:09:26.472657 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5"
format_stamp: "Formatted at 2026-06-20 09:09:26 on dist-test-slave-92t4"
server_key: "682b18aeef495e213cec000472da40fd"
server_key_iv: "95de0637515afae9145058b1a2c7b513"
server_key_version: "encryptionkey@0"
I20260620 09:09:26.476392 20976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:09:26.478942 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:26.480263 20976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:26.480400 20976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5"
format_stamp: "Formatted at 2026-06-20 09:09:26 on dist-test-slave-92t4"
server_key: "682b18aeef495e213cec000472da40fd"
server_key_iv: "95de0637515afae9145058b1a2c7b513"
server_key_version: "encryptionkey@0"
I20260620 09:09:26.480525 20976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:26.506762 20976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:26.510491 20976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:26.510739 20976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:26.518193 20976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37697
I20260620 09:09:26.518214 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37697 every 8 connection(s)
I20260620 09:09:26.519196 20976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:26.520771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 20976
I20260620 09:09:26.520920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:26.521255 392 external_mini_cluster.cc:1428] Setting key 42013284c563740b16c62a2e58f06ad7
I20260620 09:09:26.522457 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:26.529570 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: Bootstrap starting.
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[20963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:09:26.532250 21044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:26.533259 21044 log.cc:824] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:26.535604 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: No bootstrap required, opened a new log
I20260620 09:09:26.536959 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:26.522758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56933 (local address 127.0.0.1:37697)
0620 09:09:26.523209 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:26.523230 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:26.523274 (+ 44us) server_negotiation.cc:407] Connection header received
0620 09:09:26.524167 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:26.524190 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:26.524634 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:26.525066 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:26.525795 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:26.527271 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:26.528198 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:26.528898 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:26.529093 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:26.531836 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:26.531843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:26.531859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:26.531891 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:26.533903 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:26.534350 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:26.534357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:26.534362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:26.534493 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:26.534741 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:26.534744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:26.534746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:26.535308 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:26.535494 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:26.535704 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 09:09:26.536021 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260620 09:09:26.540027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:26.540009 21044 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260620 09:09:26.540267 21044 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:26.540355 21044 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02af1b7c1e348a4a70ffe4792d5e6f5, State: Initialized, Role: FOLLOWER
I20260620 09:09:26.540797 21044 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260620 09:09:26.540920 21044 raft_consensus.cc:388] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:26.541003 21044 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:26.541132 21044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:26.542140 21044 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260620 09:09:26.542562 21044 leader_election.cc:302] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f02af1b7c1e348a4a70ffe4792d5e6f5; no voters:
I20260620 09:09:26.542888 21044 leader_election.cc:288] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:26.543013 21054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:26.543259 21054 raft_consensus.cc:686] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [term 1 LEADER]: Becoming Leader. State: Replica: f02af1b7c1e348a4a70ffe4792d5e6f5, State: Running, Role: LEADER
I20260620 09:09:26.543615 21054 consensus_queue.cc:237] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260620 09:09:26.544198 21044 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:26.545223 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } } }
I20260620 09:09:26.545347 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:26.545594 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02af1b7c1e348a4a70ffe4792d5e6f5. Latest consensus state: current_term: 1 leader_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02af1b7c1e348a4a70ffe4792d5e6f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } } }
I20260620 09:09:26.545682 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5 [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:26.550071 21069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:26.550161 21069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:26.550222 21063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:26.551235 21063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:26.552130 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:26.542349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49041 (local address 127.0.0.1:37697)
0620 09:09:26.542480 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:26.542485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:26.542667 (+ 182us) server_negotiation.cc:407] Connection header received
0620 09:09:26.542797 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:26.542803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:26.542866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:26.542953 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:26.543846 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:26.544622 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:26.546211 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:26.546497 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:26.546588 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:26.548097 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:26.548103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:26.548107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:26.548131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:26.549924 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:26.550443 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:26.550447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:26.550449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:26.550537 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:26.550850 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:26.550853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:26.550855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:26.551077 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:26.551200 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:26.551862 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 09:09:26.551980 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:09:26.557839 21063 catalog_manager.cc:1347] Generated new cluster ID: 651d4ce53de64ea29eaca8d0c8333ff7
I20260620 09:09:26.557920 21063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:26.573755 21063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:26.575273 21063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:26.589498 21063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f02af1b7c1e348a4a70ffe4792d5e6f5: Generated new TSK 0
I20260620 09:09:26.590268 21063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:26.596222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 20976
2026-06-20T09:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[21075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[21075](info): set up 2 sockets
Jun 20 09:09:26 dist-test-slave-92t4 krb5kdc[21075](info): commencing operation
krb5kdc: starting...
W20260620 09:09:27.629323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:27 dist-test-slave-92t4 krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:27.752269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:27.864326 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:27.864665 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:27.864769 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:27.868818 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:27.868929 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:27.868965 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:27.869007 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:27.869059 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:27.874466 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:27.875602 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:27.876672 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:27.885500 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:27.885594 21088 server_base.cc:1034] running on GCE node
W20260620 09:09:27.885502 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:27.885505 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:27.886377 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:27.887538 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:27.888742 21088 hybrid_clock.cc:648] HybridClock initialized: now 1781946567888725 us; error 30 us; skew 500 ppm
Jun 20 09:09:27 dist-test-slave-92t4 krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:27.891611 21088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:27.892612 21088 webserver.cc:460] Webserver started at http://127.0.0.1:33883/ using document root <none> and password file <none>
I20260620 09:09:27.893246 21088 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:27.893344 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:27.893589 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:27.895673 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "5bfcb364b5a244008355c5f76bcbcbad"
format_stamp: "Formatted at 2026-06-20 09:09:27 on dist-test-slave-92t4"
server_key: "9e2e1a57a847f994ec1fef2197b237b3"
server_key_iv: "e72b541ca223c9eb9875c70e127036a5"
server_key_version: "encryptionkey@0"
I20260620 09:09:27.896251 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bfcb364b5a244008355c5f76bcbcbad"
format_stamp: "Formatted at 2026-06-20 09:09:27 on dist-test-slave-92t4"
server_key: "9e2e1a57a847f994ec1fef2197b237b3"
server_key_iv: "e72b541ca223c9eb9875c70e127036a5"
server_key_version: "encryptionkey@0"
I20260620 09:09:27.900115 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:27.902736 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:27.904008 21088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 09:09:27.904166 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "5bfcb364b5a244008355c5f76bcbcbad"
format_stamp: "Formatted at 2026-06-20 09:09:27 on dist-test-slave-92t4"
server_key: "9e2e1a57a847f994ec1fef2197b237b3"
server_key_iv: "e72b541ca223c9eb9875c70e127036a5"
server_key_version: "encryptionkey@0"
I20260620 09:09:27.904289 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:27.930089 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:27.933421 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:27.933655 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:27.941213 21088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43789
I20260620 09:09:27.941236 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43789 every 8 connection(s)
I20260620 09:09:27.942253 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:27.945139 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:27.949156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21088
I20260620 09:09:27.949334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:27.949623 392 external_mini_cluster.cc:1428] Setting key b404307d826dd3bec635c50bbd981d99
I20260620 09:09:27.951363 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: Bootstrap starting.
I20260620 09:09:27.953801 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:27.954770 21156 log.cc:824] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:27.956707 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: No bootstrap required, opened a new log
I20260620 09:09:27.959010 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:27.951082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47119 (local address 127.0.0.1:43789)
0620 09:09:27.951580 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:27.951599 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:27.951636 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:09:27.952306 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:27.952324 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:27.952690 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:27.953080 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:27.953515 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:27.954559 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:27.955527 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:27.956006 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:27.956179 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:27.957107 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:27.957113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:27.957127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:27.957538 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:27.957591 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:27.957614 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:27.957727 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:27.957845 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 09:09:27.958069 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260620 09:09:27.960224 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } }
I20260620 09:09:27.960486 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:27.960563 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfcb364b5a244008355c5f76bcbcbad, State: Initialized, Role: FOLLOWER
I20260620 09:09:27.961097 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } }
I20260620 09:09:27.961264 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:27.961334 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:27.961431 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:27.962177 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:27.962574 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } }
I20260620 09:09:27.962955 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bfcb364b5a244008355c5f76bcbcbad; no voters:
I20260620 09:09:27.963284 21156 leader_election.cc:288] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:27.963413 21161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:27.963712 21161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [term 1 LEADER]: Becoming Leader. State: Replica: 5bfcb364b5a244008355c5f76bcbcbad, State: Running, Role: LEADER
I20260620 09:09:27.964092 21161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } }
I20260620 09:09:27.964768 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:27.965492 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bfcb364b5a244008355c5f76bcbcbad. Latest consensus state: current_term: 1 leader_uuid: "5bfcb364b5a244008355c5f76bcbcbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } } }
I20260620 09:09:27.965644 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:27.966315 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bfcb364b5a244008355c5f76bcbcbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bfcb364b5a244008355c5f76bcbcbad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43789 } } }
I20260620 09:09:27.966415 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:27.969506 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:27.964602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43747 (local address 127.0.0.1:43789)
0620 09:09:27.964711 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:27.964731 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:27.965190 (+ 459us) server_negotiation.cc:407] Connection header received
0620 09:09:27.965335 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:27.965339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:27.965390 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:27.965476 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:27.965812 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:27.966643 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:27.967804 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:27.968091 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:27.968176 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:27.969028 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:27.969032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:27.969035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:27.969077 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:27.969112 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:27.969126 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:27.969200 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:27.969305 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:09:27.969364 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 09:09:27.970669 21181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:27.970732 21181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:27.970806 21174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:27.971505 21174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:27.977097 21174 catalog_manager.cc:1347] Generated new cluster ID: 57c7f29ee50d4d498aefd9a7ede53d1e
I20260620 09:09:27.977169 21174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:27.997849 21174 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:27.998847 21174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:28.008879 21174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bfcb364b5a244008355c5f76bcbcbad: Generated new TSK 0
I20260620 09:09:28.009650 21174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:28.020422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21088
2026-06-20T09:09:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:28 dist-test-slave-92t4 krb5kdc[21187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:28 dist-test-slave-92t4 krb5kdc[21187](info): set up 2 sockets
Jun 20 09:09:28 dist-test-slave-92t4 krb5kdc[21187](info): commencing operation
krb5kdc: starting...
W20260620 09:09:29.061404 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:29 dist-test-slave-92t4 krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:29.188089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:29.307154 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:29.307492 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:29.307598 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:29.311798 21200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:29.311903 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:29.311942 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:29.312000 21200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:29.312037 21200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:29.318035 21200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:29.319244 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:29.320389 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:29.329412 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:29.329403 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:29.329403 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:29.329576 21200 server_base.cc:1034] running on GCE node
I20260620 09:09:29.330303 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:29.331496 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:29.332667 21200 hybrid_clock.cc:648] HybridClock initialized: now 1781946569332649 us; error 33 us; skew 500 ppm
Jun 20 09:09:29 dist-test-slave-92t4 krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:29.335513 21200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:29.336513 21200 webserver.cc:460] Webserver started at http://127.0.0.1:46135/ using document root <none> and password file <none>
I20260620 09:09:29.337316 21200 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:29.337390 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:29.337661 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:29.339998 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "3e101fb427704854b3f7bebd2a442a4e"
format_stamp: "Formatted at 2026-06-20 09:09:29 on dist-test-slave-92t4"
server_key: "53b11080a45f67f123fc57d74324b6cf"
server_key_iv: "f8574d09e43a34ec30eea4beaf62cd76"
server_key_version: "encryptionkey@0"
I20260620 09:09:29.340598 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e101fb427704854b3f7bebd2a442a4e"
format_stamp: "Formatted at 2026-06-20 09:09:29 on dist-test-slave-92t4"
server_key: "53b11080a45f67f123fc57d74324b6cf"
server_key_iv: "f8574d09e43a34ec30eea4beaf62cd76"
server_key_version: "encryptionkey@0"
I20260620 09:09:29.344524 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 09:09:29.347105 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:29.348414 21200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:29.348558 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "3e101fb427704854b3f7bebd2a442a4e"
format_stamp: "Formatted at 2026-06-20 09:09:29 on dist-test-slave-92t4"
server_key: "53b11080a45f67f123fc57d74324b6cf"
server_key_iv: "f8574d09e43a34ec30eea4beaf62cd76"
server_key_version: "encryptionkey@0"
I20260620 09:09:29.348678 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:29.353477 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:29.356735 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:29.356932 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:29.364264 21200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44795
I20260620 09:09:29.364295 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44795 every 8 connection(s)
I20260620 09:09:29.365370 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:29.368292 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:29.373945 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e: Bootstrap starting.
I20260620 09:09:29.375154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21200
I20260620 09:09:29.375311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:29.375609 392 external_mini_cluster.cc:1428] Setting key 799b3aaa8e754ddb09d67dfd690e9ce5
I20260620 09:09:29.376917 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:29.377738 21268 log.cc:824] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:29.379773 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e: No bootstrap required, opened a new log
I20260620 09:09:29.380863 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:29.376794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41443 (local address 127.0.0.1:44795)
0620 09:09:29.377289 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:29.377304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:29.377342 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:09:29.378061 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:29.378107 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:29.378488 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:29.378847 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:29.379664 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:29.379669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:29.379681 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:29.379691 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:29.379745 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:29.379764 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:29.379888 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:29.379993 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 09:09:29.380157 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":188,"threads_started":1}
I20260620 09:09:29.384191 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260620 09:09:29.384447 21268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:29.384550 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e101fb427704854b3f7bebd2a442a4e, State: Initialized, Role: FOLLOWER
I20260620 09:09:29.384692 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:29.385120 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260620 09:09:29.385288 21268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:29.385358 21268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:29.385456 21268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:29.386515 21268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260620 09:09:29.386898 21268 leader_election.cc:302] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e101fb427704854b3f7bebd2a442a4e; no voters:
I20260620 09:09:29.387272 21268 leader_election.cc:288] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:29.387398 21278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:29.387641 21278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [term 1 LEADER]: Becoming Leader. State: Replica: 3e101fb427704854b3f7bebd2a442a4e, State: Running, Role: LEADER
I20260620 09:09:29.388068 21278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } }
I20260620 09:09:29.388656 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:29.389066 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:29.386950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37193 (local address 127.0.0.1:44795)
0620 09:09:29.387094 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:29.387098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:29.387326 (+ 228us) server_negotiation.cc:407] Connection header received
0620 09:09:29.387493 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:29.387497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:29.387541 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:29.387632 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:29.388484 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:29.388488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:29.388490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:29.388502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:29.388530 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:29.388542 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:29.388616 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:29.388885 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 09:09:29.388924 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:29.389901 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e101fb427704854b3f7bebd2a442a4e. Latest consensus state: current_term: 1 leader_uuid: "3e101fb427704854b3f7bebd2a442a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } } }
I20260620 09:09:29.390039 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:29.390450 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e101fb427704854b3f7bebd2a442a4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e101fb427704854b3f7bebd2a442a4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44795 } } }
I20260620 09:09:29.390568 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:29.390923 21287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:29.394058 21287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:29.400166 21287 catalog_manager.cc:1347] Generated new cluster ID: 8c345f51a2fd4f8791b46768e65adc07
I20260620 09:09:29.400256 21287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:29.415546 21287 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:29.415915 21287 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:29.434073 21287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e101fb427704854b3f7bebd2a442a4e: Generated new TSK 0
I20260620 09:09:29.434912 21287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:29.444137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21200
2026-06-20T09:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:29 dist-test-slave-92t4 krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:29 dist-test-slave-92t4 krb5kdc[21299](info): set up 2 sockets
Jun 20 09:09:29 dist-test-slave-92t4 krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260620 09:09:30.481436 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:30.608019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:30.721300 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:30.721599 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:30.721642 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:30.725845 21312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:30.725919 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:30.725939 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:30.725955 21312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:30.725970 21312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:30.731721 21312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:30.732892 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:30.734083 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:30.743290 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:30.743321 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:30.743290 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:30.743583 21312 server_base.cc:1034] running on GCE node
I20260620 09:09:30.744277 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:30.745563 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:30.746765 21312 hybrid_clock.cc:648] HybridClock initialized: now 1781946570746730 us; error 31 us; skew 500 ppm
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:30.753525 21312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:30.754623 21312 webserver.cc:460] Webserver started at http://127.0.0.1:43175/ using document root <none> and password file <none>
I20260620 09:09:30.755240 21312 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:30.755335 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:30.755580 21312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:30.757985 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "2a50437bb3fa44c49f64a702cefb9c8d"
format_stamp: "Formatted at 2026-06-20 09:09:30 on dist-test-slave-92t4"
server_key: "08645848f2ef491ab85e44a766e97d86"
server_key_iv: "c69ee242f232d1fe8856533b044d90d4"
server_key_version: "encryptionkey@0"
I20260620 09:09:30.758733 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a50437bb3fa44c49f64a702cefb9c8d"
format_stamp: "Formatted at 2026-06-20 09:09:30 on dist-test-slave-92t4"
server_key: "08645848f2ef491ab85e44a766e97d86"
server_key_iv: "c69ee242f232d1fe8856533b044d90d4"
server_key_version: "encryptionkey@0"
I20260620 09:09:30.762673 21312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:30.765348 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:30.767038 21312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 09:09:30.767180 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "2a50437bb3fa44c49f64a702cefb9c8d"
format_stamp: "Formatted at 2026-06-20 09:09:30 on dist-test-slave-92t4"
server_key: "08645848f2ef491ab85e44a766e97d86"
server_key_iv: "c69ee242f232d1fe8856533b044d90d4"
server_key_version: "encryptionkey@0"
I20260620 09:09:30.767333 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:30.779770 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:30.783270 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:30.783496 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:30.791031 21312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33201
I20260620 09:09:30.791054 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33201 every 8 connection(s)
I20260620 09:09:30.792039 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:30.794595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21312
I20260620 09:09:30.794755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:30.795053 392 external_mini_cluster.cc:1428] Setting key 224e7262d8c5633092746e8d4cc357ac
I20260620 09:09:30.795073 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:30.801573 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: Bootstrap starting.
I20260620 09:09:30.803968 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: Neither blocks nor log segments found. Creating new log.
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:09:30.804981 21380 log.cc:824] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:30.807331 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: No bootstrap required, opened a new log
I20260620 09:09:30.810101 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:30.796557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:43783 (local address 127.0.0.1:33201)
0620 09:09:30.796991 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:30.797007 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:30.797105 (+ 98us) server_negotiation.cc:407] Connection header received
0620 09:09:30.797870 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:30.797889 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:30.798238 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:30.798584 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:30.799410 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:30.800442 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:30.801381 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:30.801926 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:30.802099 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:30.805019 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:30.805024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:30.805097 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:30.805159 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:30.807017 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:30.807603 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:30.807609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:30.807614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:30.807728 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:30.808091 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:30.808099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:30.808101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:30.808535 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:30.808761 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:30.809128 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 09:09:30.809355 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260620 09:09:30.811451 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 09:09:30.811699 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:30.811792 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a50437bb3fa44c49f64a702cefb9c8d, State: Initialized, Role: FOLLOWER
I20260620 09:09:30.812335 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 09:09:30.812505 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:30.812572 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:30.812660 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:30.813473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:30.813848 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 09:09:30.814199 21380 leader_election.cc:302] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a50437bb3fa44c49f64a702cefb9c8d; no voters:
I20260620 09:09:30.814527 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:30.814653 21385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:30.814893 21385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [term 1 LEADER]: Becoming Leader. State: Replica: 2a50437bb3fa44c49f64a702cefb9c8d, State: Running, Role: LEADER
I20260620 09:09:30.815287 21385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260620 09:09:30.815889 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:30.816625 21390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a50437bb3fa44c49f64a702cefb9c8d. Latest consensus state: current_term: 1 leader_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260620 09:09:30.817420 21390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:30.817260 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a50437bb3fa44c49f64a702cefb9c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260620 09:09:30.817714 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:30.821080 21405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:30.821178 21405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:30.821290 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:30.822229 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:30.825337 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:30.815967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:60961 (local address 127.0.0.1:33201)
0620 09:09:30.816107 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:30.816112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:30.816335 (+ 223us) server_negotiation.cc:407] Connection header received
0620 09:09:30.816550 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:30.816553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:30.816605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:30.816662 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:30.817717 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:30.818463 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:30.819399 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:30.819733 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:30.819853 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:30.821602 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:30.821606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:30.821608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:30.821633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:30.823400 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:30.823964 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:30.823968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:30.823969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:30.824055 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:30.824427 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:30.824430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:30.824432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:30.824611 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:30.824779 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:30.825082 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 09:09:30.825178 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:09:30.828454 21401 catalog_manager.cc:1347] Generated new cluster ID: 8841b2aacd4c4f96b4146604ec0cd0cc
I20260620 09:09:30.828518 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:30.857095 21401 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:30.858397 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:30.877902 21401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a50437bb3fa44c49f64a702cefb9c8d: Generated new TSK 0
I20260620 09:09:30.878712 21401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:30.885025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21312
W20260620 09:09:30.891026 21297 connection.cc:535] client connection to 127.0.0.1:33201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33201): Connection reset by peer (error 104)
2026-06-20T09:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21411](info): set up 2 sockets
Jun 20 09:09:30 dist-test-slave-92t4 krb5kdc[21411](info): commencing operation
krb5kdc: starting...
W20260620 09:09:31.945456 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:32 dist-test-slave-92t4 krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:32.074389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:32.191185 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:32.191529 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:32.191613 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:32.195767 21424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:32.195873 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:32.195911 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:32.195969 21424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:32.196009 21424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:32.201562 21424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:32.202781 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:32.203919 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:32.212968 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:32.212955 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:32.213014 21424 server_base.cc:1034] running on GCE node
W20260620 09:09:32.212960 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:32.213917 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:32.215061 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:32.216241 21424 hybrid_clock.cc:648] HybridClock initialized: now 1781946572216218 us; error 28 us; skew 500 ppm
Jun 20 09:09:32 dist-test-slave-92t4 krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:32.219099 21424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:32.220047 21424 webserver.cc:460] Webserver started at http://127.0.0.1:39929/ using document root <none> and password file <none>
I20260620 09:09:32.220687 21424 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:32.220750 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:32.220997 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:32.223206 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "8c78303930ef4531af009ac65e34aa55"
format_stamp: "Formatted at 2026-06-20 09:09:32 on dist-test-slave-92t4"
server_key: "eb75c2d588dcb32fee21fb7ae7726084"
server_key_iv: "6373278f4fda6c65def7b2117cd049bc"
server_key_version: "encryptionkey@0"
I20260620 09:09:32.223740 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c78303930ef4531af009ac65e34aa55"
format_stamp: "Formatted at 2026-06-20 09:09:32 on dist-test-slave-92t4"
server_key: "eb75c2d588dcb32fee21fb7ae7726084"
server_key_iv: "6373278f4fda6c65def7b2117cd049bc"
server_key_version: "encryptionkey@0"
I20260620 09:09:32.227576 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:32.230108 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:32.231441 21424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:32.231593 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "8c78303930ef4531af009ac65e34aa55"
format_stamp: "Formatted at 2026-06-20 09:09:32 on dist-test-slave-92t4"
server_key: "eb75c2d588dcb32fee21fb7ae7726084"
server_key_iv: "6373278f4fda6c65def7b2117cd049bc"
server_key_version: "encryptionkey@0"
I20260620 09:09:32.231755 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:32.248208 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:32.251503 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:32.251718 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:32.259052 21424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34105
I20260620 09:09:32.259068 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34105 every 8 connection(s)
I20260620 09:09:32.260049 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:32.261206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21424
I20260620 09:09:32.261355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:32.261653 392 external_mini_cluster.cc:1428] Setting key c15fe8ffa2f69905c40bd150cd584aae
I20260620 09:09:32.263402 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:32.270143 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55: Bootstrap starting.
I20260620 09:09:32.271628 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:32.262946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:52303 (local address 127.0.0.1:34105)
0620 09:09:32.263366 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:32.263386 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:32.263429 (+ 43us) server_negotiation.cc:407] Connection header received
0620 09:09:32.264237 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:32.264259 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:32.264705 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:32.265086 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:32.265493 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:32.267019 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:32.267959 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:32.268646 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:32.268816 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:32.269685 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:32.269691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:32.269706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:32.270182 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:32.270252 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:32.270282 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:32.270419 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:32.270585 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 09:09:32.270713 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260620 09:09:32.277746 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:32.278710 21492 log.cc:824] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:32.280268 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:32.280922 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55: No bootstrap required, opened a new log
I20260620 09:09:32.285871 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20260620 09:09:32.286233 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:32.286319 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c78303930ef4531af009ac65e34aa55, State: Initialized, Role: FOLLOWER
I20260620 09:09:32.286942 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20260620 09:09:32.287137 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:32.282428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:56751 (local address 127.0.0.1:34105)
0620 09:09:32.282576 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:32.282581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:32.282770 (+ 189us) server_negotiation.cc:407] Connection header received
0620 09:09:32.282920 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:32.282922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:32.282965 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:32.283024 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:32.283387 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:32.283922 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:32.285152 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:32.285377 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:32.285460 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:32.286492 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:32.286495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:32.286497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:32.286511 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:32.286538 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:32.286549 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:32.286621 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:32.286849 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 09:09:32.286923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 09:09:32.287200 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:32.287304 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:32.287427 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:32.288568 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20260620 09:09:32.288885 21492 leader_election.cc:302] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c78303930ef4531af009ac65e34aa55; no voters:
I20260620 09:09:32.289225 21492 leader_election.cc:288] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:32.289342 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:32.289604 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [term 1 LEADER]: Becoming Leader. State: Replica: 8c78303930ef4531af009ac65e34aa55, State: Running, Role: LEADER
I20260620 09:09:32.289971 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } }
I20260620 09:09:32.290550 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:32.291728 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c78303930ef4531af009ac65e34aa55. Latest consensus state: current_term: 1 leader_uuid: "8c78303930ef4531af009ac65e34aa55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } } }
I20260620 09:09:32.291728 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c78303930ef4531af009ac65e34aa55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78303930ef4531af009ac65e34aa55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34105 } } }
I20260620 09:09:32.291913 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:32.291960 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:32.292361 21512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:32.295934 21512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:32.301579 21512 catalog_manager.cc:1347] Generated new cluster ID: f6544d7a787c4a5f9d7859f92fb4b8b4
I20260620 09:09:32.301659 21512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:32.308682 21512 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:32.309752 21512 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:32.320330 21512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c78303930ef4531af009ac65e34aa55: Generated new TSK 0
I20260620 09:09:32.321142 21512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:32.327430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21424
W20260620 09:09:32.332862 21409 connection.cc:535] client connection to 127.0.0.1:34105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34105): Connection reset by peer (error 104)
2026-06-20T09:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:32 dist-test-slave-92t4 krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:32 dist-test-slave-92t4 krb5kdc[21523](info): set up 2 sockets
Jun 20 09:09:32 dist-test-slave-92t4 krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260620 09:09:33.365398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:33 dist-test-slave-92t4 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:33.492324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:33.606525 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:33.606926 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:33.607008 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:33.611279 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:33.611389 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:33.611428 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:33.611488 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:33.611543 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:33.617369 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:33.618599 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:33.619704 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:33.628623 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:33.628615 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:33.628615 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:33.628947 21536 server_base.cc:1034] running on GCE node
I20260620 09:09:33.629690 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:33.630793 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:33.631963 21536 hybrid_clock.cc:648] HybridClock initialized: now 1781946573631943 us; error 34 us; skew 500 ppm
Jun 20 09:09:33 dist-test-slave-92t4 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:33.634896 21536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:33.635913 21536 webserver.cc:460] Webserver started at http://127.0.0.1:34267/ using document root <none> and password file <none>
I20260620 09:09:33.636507 21536 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:33.636570 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:33.636790 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:33.638993 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad"
format_stamp: "Formatted at 2026-06-20 09:09:33 on dist-test-slave-92t4"
server_key: "e7e373144d67dac887af5bb965d64737"
server_key_iv: "04e19d40a4388369c55010cdb4b02df1"
server_key_version: "encryptionkey@0"
I20260620 09:09:33.639603 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad"
format_stamp: "Formatted at 2026-06-20 09:09:33 on dist-test-slave-92t4"
server_key: "e7e373144d67dac887af5bb965d64737"
server_key_iv: "04e19d40a4388369c55010cdb4b02df1"
server_key_version: "encryptionkey@0"
I20260620 09:09:33.643771 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 09:09:33.646842 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:33.648360 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 09:09:33.648551 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad"
format_stamp: "Formatted at 2026-06-20 09:09:33 on dist-test-slave-92t4"
server_key: "e7e373144d67dac887af5bb965d64737"
server_key_iv: "04e19d40a4388369c55010cdb4b02df1"
server_key_version: "encryptionkey@0"
I20260620 09:09:33.648700 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:33.653677 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:33.656754 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:33.656960 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:33.664346 21536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36045
I20260620 09:09:33.664352 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36045 every 8 connection(s)
I20260620 09:09:33.665369 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:33.668435 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:33.669029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21536
I20260620 09:09:33.669201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:33.669523 392 external_mini_cluster.cc:1428] Setting key cdc9593e674df0e2ad8571934ffc6d1d
I20260620 09:09:33.674892 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:33.670952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:40277 (local address 127.0.0.1:36045)
0620 09:09:33.671406 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:33.671420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:33.671454 (+ 34us) server_negotiation.cc:407] Connection header received
0620 09:09:33.672145 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:33.672162 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:33.672452 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:33.672828 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:33.673664 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:33.673670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:33.673691 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:33.673702 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:33.673761 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:33.673780 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:33.673936 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:33.674006 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 09:09:33.674174 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260620 09:09:33.675585 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad: Bootstrap starting.
I20260620 09:09:33.677970 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:33.678411 21604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:33.679191 21604 log.cc:824] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:33.681713 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad: No bootstrap required, opened a new log
I20260620 09:09:33.681890 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:33.680105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:44359 (local address 127.0.0.1:36045)
0620 09:09:33.680235 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:33.680240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:33.680437 (+ 197us) server_negotiation.cc:407] Connection header received
0620 09:09:33.680564 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:33.680566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:33.680597 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:33.680663 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:33.681442 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:33.681445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:33.681447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:33.681456 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:33.681481 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:33.681491 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:33.681561 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:33.681665 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:09:33.681696 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:09:33.685779 21604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260620 09:09:33.686051 21604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:33.686124 21604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca2f2a3b6ba49c3bd28ec50ca53b4ad, State: Initialized, Role: FOLLOWER
I20260620 09:09:33.686640 21604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260620 09:09:33.686770 21604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:33.686857 21604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:33.686980 21604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:33.688040 21604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260620 09:09:33.688442 21604 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca2f2a3b6ba49c3bd28ec50ca53b4ad; no voters:
I20260620 09:09:33.688871 21604 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:33.688953 21614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:33.689225 21614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [term 1 LEADER]: Becoming Leader. State: Replica: 2ca2f2a3b6ba49c3bd28ec50ca53b4ad, State: Running, Role: LEADER
I20260620 09:09:33.689538 21614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260620 09:09:33.690387 21604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:33.691383 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca2f2a3b6ba49c3bd28ec50ca53b4ad. Latest consensus state: current_term: 1 leader_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260620 09:09:33.691511 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:33.691883 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca2f2a3b6ba49c3bd28ec50ca53b4ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260620 09:09:33.691939 21623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:33.692030 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:33.695716 21623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:33.702057 21623 catalog_manager.cc:1347] Generated new cluster ID: f7ac3369378c48848c6fff7f473a060d
I20260620 09:09:33.702167 21623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:33.710242 21623 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:33.710598 21623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:33.725087 21623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca2f2a3b6ba49c3bd28ec50ca53b4ad: Generated new TSK 0
I20260620 09:09:33.726074 21623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:33.729415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21536
2026-06-20T09:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:33 dist-test-slave-92t4 krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:33 dist-test-slave-92t4 krb5kdc[21635](info): set up 2 sockets
Jun 20 09:09:33 dist-test-slave-92t4 krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260620 09:09:34.773337 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:34 dist-test-slave-92t4 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:34.897914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:35.030854 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:35.031152 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:35.031198 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:35.035315 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:35.035393 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:35.035414 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:35.035431 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:35.035446 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:35.041168 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:35.042275 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:35.043295 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:35.052582 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:35.052651 21648 server_base.cc:1034] running on GCE node
W20260620 09:09:35.052591 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:35.052582 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:35.053630 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:35.054929 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:35.056100 21648 hybrid_clock.cc:648] HybridClock initialized: now 1781946575056086 us; error 38 us; skew 500 ppm
Jun 20 09:09:35 dist-test-slave-92t4 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:35.059131 21648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:35.060132 21648 webserver.cc:460] Webserver started at http://127.0.0.1:42989/ using document root <none> and password file <none>
I20260620 09:09:35.060725 21648 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:35.060791 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:35.061017 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:35.063287 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "c5865a460bb148df841bbc6a598aedf6"
format_stamp: "Formatted at 2026-06-20 09:09:35 on dist-test-slave-92t4"
server_key: "3e1d0162b1d8518e9b2f4df57b52a3f1"
server_key_iv: "8901be1a28e2bbd7ecd96028541ae76a"
server_key_version: "encryptionkey@0"
I20260620 09:09:35.063926 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5865a460bb148df841bbc6a598aedf6"
format_stamp: "Formatted at 2026-06-20 09:09:35 on dist-test-slave-92t4"
server_key: "3e1d0162b1d8518e9b2f4df57b52a3f1"
server_key_iv: "8901be1a28e2bbd7ecd96028541ae76a"
server_key_version: "encryptionkey@0"
I20260620 09:09:35.068143 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:35.070813 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:35.072140 21648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:35.072304 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "c5865a460bb148df841bbc6a598aedf6"
format_stamp: "Formatted at 2026-06-20 09:09:35 on dist-test-slave-92t4"
server_key: "3e1d0162b1d8518e9b2f4df57b52a3f1"
server_key_iv: "8901be1a28e2bbd7ecd96028541ae76a"
server_key_version: "encryptionkey@0"
I20260620 09:09:35.072443 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:35.086997 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:35.090533 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:35.090770 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:35.098347 21648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45527
I20260620 09:09:35.098381 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45527 every 8 connection(s)
I20260620 09:09:35.099416 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:35.102510 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:35.105199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21648
I20260620 09:09:35.105357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:35.105670 392 external_mini_cluster.cc:1428] Setting key 14372b489bf27ba4b10567df517889db
I20260620 09:09:35.109129 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: Bootstrap starting.
I20260620 09:09:35.111450 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:35.112210 21716 log.cc:824] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:35.114403 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: No bootstrap required, opened a new log
Jun 20 09:09:35 dist-test-slave-92t4 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 09:09:35.118487 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 09:09:35.118758 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:35.118852 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5865a460bb148df841bbc6a598aedf6, State: Initialized, Role: FOLLOWER
I20260620 09:09:35.119385 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 09:09:35.119578 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:35.119666 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:35.119781 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:35.120752 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 09:09:35.120978 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:35.107196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:55697 (local address 127.0.0.1:45527)
0620 09:09:35.107675 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:35.107691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:35.107728 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:09:35.108394 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:35.108412 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:35.108781 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:35.109180 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:35.110008 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:35.111032 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:35.112021 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:35.112576 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:35.112764 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:35.115533 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:35.115539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:35.115555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:35.115587 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:35.117468 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:35.118028 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:35.118033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:35.118038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:35.118155 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:35.118514 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:35.118520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:35.118526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:35.119053 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:35.119281 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:35.119707 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 09:09:35.120140 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260620 09:09:35.121187 21716 leader_election.cc:302] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5865a460bb148df841bbc6a598aedf6; no voters:
I20260620 09:09:35.121588 21716 leader_election.cc:288] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:35.121665 21721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:35.121867 21721 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [term 1 LEADER]: Becoming Leader. State: Replica: c5865a460bb148df841bbc6a598aedf6, State: Running, Role: LEADER
I20260620 09:09:35.122220 21721 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 09:09:35.123098 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:35.124404 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5865a460bb148df841bbc6a598aedf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260620 09:09:35.124622 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:35.124822 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:35.125679 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5865a460bb148df841bbc6a598aedf6. Latest consensus state: current_term: 1 leader_uuid: "c5865a460bb148df841bbc6a598aedf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5865a460bb148df841bbc6a598aedf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260620 09:09:35.125808 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6 [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:35.129119 21737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:35.129208 21737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:35.129267 21731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:35.130173 21731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:35.136678 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:35.127287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:54403 (local address 127.0.0.1:45527)
0620 09:09:35.127411 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:35.127416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:35.127713 (+ 297us) server_negotiation.cc:407] Connection header received
0620 09:09:35.127842 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:35.127845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:35.127894 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:35.127987 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:35.128748 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:35.129477 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:35.130451 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:35.130702 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:35.130824 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:35.132278 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:35.132282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:35.132284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:35.132310 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:35.134408 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:35.135072 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:35.135076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:35.135078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:35.135173 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:35.135609 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:35.135615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:35.135619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:35.135836 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:35.136001 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:35.136283 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 09:09:35.136381 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 09:09:35.137498 21731 catalog_manager.cc:1347] Generated new cluster ID: 51a787a2b56b4e14948b995b514761a9
I20260620 09:09:35.137566 21731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:35.149469 21731 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:35.150561 21731 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:35.160048 21731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5865a460bb148df841bbc6a598aedf6: Generated new TSK 0
I20260620 09:09:35.160800 21731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:35.168655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21648
W20260620 09:09:35.174283 21633 connection.cc:535] client connection to 127.0.0.1:45527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45527): Connection reset by peer (error 104)
2026-06-20T09:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:35 dist-test-slave-92t4 krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:35 dist-test-slave-92t4 krb5kdc[21747](info): set up 2 sockets
Jun 20 09:09:35 dist-test-slave-92t4 krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260620 09:09:36.201439 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:36 dist-test-slave-92t4 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:36.325323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:36.443200 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:36.443507 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:36.443552 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:36.447840 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:36.447930 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:36.447952 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:36.447969 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:36.447985 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:36.453842 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:36.455045 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:36.456146 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:36.465276 21760 server_base.cc:1034] running on GCE node
W20260620 09:09:36.465195 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:36.465243 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:36.465219 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:36.466048 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:36.467185 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:36.468350 21760 hybrid_clock.cc:648] HybridClock initialized: now 1781946576468336 us; error 27 us; skew 500 ppm
Jun 20 09:09:36 dist-test-slave-92t4 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:36.492453 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:36.493861 21760 webserver.cc:460] Webserver started at http://127.0.0.1:34793/ using document root <none> and password file <none>
I20260620 09:09:36.494525 21760 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:36.494628 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:36.494933 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:36.497264 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "8537f66c31944c31ae9e235a098196e3"
format_stamp: "Formatted at 2026-06-20 09:09:36 on dist-test-slave-92t4"
server_key: "f710ab1f74ff30b32cf038adfe064296"
server_key_iv: "2882b428a05bab25a1d8f653d5ebad26"
server_key_version: "encryptionkey@0"
I20260620 09:09:36.497871 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "8537f66c31944c31ae9e235a098196e3"
format_stamp: "Formatted at 2026-06-20 09:09:36 on dist-test-slave-92t4"
server_key: "f710ab1f74ff30b32cf038adfe064296"
server_key_iv: "2882b428a05bab25a1d8f653d5ebad26"
server_key_version: "encryptionkey@0"
I20260620 09:09:36.502050 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:09:36.504685 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:36.506102 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:36.506275 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "8537f66c31944c31ae9e235a098196e3"
format_stamp: "Formatted at 2026-06-20 09:09:36 on dist-test-slave-92t4"
server_key: "f710ab1f74ff30b32cf038adfe064296"
server_key_iv: "2882b428a05bab25a1d8f653d5ebad26"
server_key_version: "encryptionkey@0"
I20260620 09:09:36.506403 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:36.524507 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:36.528302 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:36.528537 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:36.536067 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33705
I20260620 09:09:36.536090 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33705 every 8 connection(s)
I20260620 09:09:36.537127 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:36.539971 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:36.542546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21760
I20260620 09:09:36.542688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:36.542981 392 external_mini_cluster.cc:1428] Setting key dd3a81355ed51a9906da1287d42c68bc
I20260620 09:09:36.546658 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3: Bootstrap starting.
I20260620 09:09:36.548954 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:36.549736 21828 log.cc:824] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:36.551942 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3: No bootstrap required, opened a new log
I20260620 09:09:36.552551 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:36.544497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:52321 (local address 127.0.0.1:33705)
0620 09:09:36.544974 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:36.544989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:36.545026 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:09:36.545791 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:36.545809 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:36.546186 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:36.546536 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:36.546963 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:36.547958 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:36.548914 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:36.549504 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:36.549707 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:36.550636 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:36.550641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:36.550654 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:36.551066 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:36.551120 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:36.551143 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:36.551249 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:36.551478 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 09:09:36.551645 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":134,"threads_started":1}
I20260620 09:09:36.555758 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:36.556181 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260620 09:09:36.556437 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:36.556510 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8537f66c31944c31ae9e235a098196e3, State: Initialized, Role: FOLLOWER
I20260620 09:09:36.556945 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260620 09:09:36.557116 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:36.557196 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:36.557301 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:36.558467 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260620 09:09:36.558876 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8537f66c31944c31ae9e235a098196e3; no voters:
I20260620 09:09:36.559204 21828 leader_election.cc:288] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:36.559316 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:36.559507 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8537f66c31944c31ae9e235a098196e3, State: Running, Role: LEADER
I20260620 09:09:36.559875 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } }
I20260620 09:09:36.560556 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:36.560994 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8537f66c31944c31ae9e235a098196e3. Latest consensus state: current_term: 1 leader_uuid: "8537f66c31944c31ae9e235a098196e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } } }
I20260620 09:09:36.561197 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:36.561470 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8537f66c31944c31ae9e235a098196e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537f66c31944c31ae9e235a098196e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33705 } } }
I20260620 09:09:36.561650 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:36.561633 21843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:36.563165 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:36.557993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:40161 (local address 127.0.0.1:33705)
0620 09:09:36.558126 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:36.558130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:36.558311 (+ 181us) server_negotiation.cc:407] Connection header received
0620 09:09:36.558465 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:36.558467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:36.558509 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:36.558604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:36.558864 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:36.559573 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:36.560733 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:36.561107 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:36.562263 (+ 1156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:36.562406 (+ 143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:36.562409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:36.562415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:36.562428 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:36.562449 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:36.562462 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:36.562950 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:36.563003 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 09:09:36.563036 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:09:36.565639 21843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:36.571898 21843 catalog_manager.cc:1347] Generated new cluster ID: 216d50e321ea4805960033a557f57e0a
I20260620 09:09:36.572016 21843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:36.586932 21843 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:36.588093 21843 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:36.600374 21843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8537f66c31944c31ae9e235a098196e3: Generated new TSK 0
I20260620 09:09:36.601270 21843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:36.613253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21760
W20260620 09:09:36.618172 21745 connection.cc:535] client connection to 127.0.0.1:33705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33705): Connection reset by peer (error 104)
2026-06-20T09:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:36 dist-test-slave-92t4 krb5kdc[21859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:36 dist-test-slave-92t4 krb5kdc[21859](info): set up 2 sockets
Jun 20 09:09:36 dist-test-slave-92t4 krb5kdc[21859](info): commencing operation
krb5kdc: starting...
W20260620 09:09:37.645509 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:37 dist-test-slave-92t4 krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 09:09:37.772701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:37.887856 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:37.888217 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:37.888294 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:37.892727 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:37.892855 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:37.892891 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:37.892930 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:37.892966 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:37.898577 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:37.899780 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:37.900864 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:37.909566 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:37.909694 21872 server_base.cc:1034] running on GCE node
W20260620 09:09:37.909559 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:37.909602 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:37.910480 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:37.911653 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:37.912839 21872 hybrid_clock.cc:648] HybridClock initialized: now 1781946577912826 us; error 29 us; skew 500 ppm
Jun 20 09:09:37 dist-test-slave-92t4 krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:37.915916 21872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 09:09:37.916843 21872 webserver.cc:460] Webserver started at http://127.0.0.1:33761/ using document root <none> and password file <none>
I20260620 09:09:37.917486 21872 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:37.917580 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:37.917827 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:37.919978 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "ed07cd205cd34a698177a2204c6b1f82"
format_stamp: "Formatted at 2026-06-20 09:09:37 on dist-test-slave-92t4"
server_key: "cde214e9abdecd149117364223db254c"
server_key_iv: "01c412137fc5d5552862d6835306376b"
server_key_version: "encryptionkey@0"
I20260620 09:09:37.920579 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed07cd205cd34a698177a2204c6b1f82"
format_stamp: "Formatted at 2026-06-20 09:09:37 on dist-test-slave-92t4"
server_key: "cde214e9abdecd149117364223db254c"
server_key_iv: "01c412137fc5d5552862d6835306376b"
server_key_version: "encryptionkey@0"
I20260620 09:09:37.924640 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:37.927577 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:37.929198 21872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:37.929364 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "ed07cd205cd34a698177a2204c6b1f82"
format_stamp: "Formatted at 2026-06-20 09:09:37 on dist-test-slave-92t4"
server_key: "cde214e9abdecd149117364223db254c"
server_key_iv: "01c412137fc5d5552862d6835306376b"
server_key_version: "encryptionkey@0"
I20260620 09:09:37.929517 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:37.934304 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:37.937897 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:37.938112 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:37.945552 21872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40693
I20260620 09:09:37.945571 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40693 every 8 connection(s)
I20260620 09:09:37.946545 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:37.949343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21872
I20260620 09:09:37.949479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:37.949548 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:37.949800 392 external_mini_cluster.cc:1428] Setting key e7c83ec381f4e73ebb3d1c6809f10f66
I20260620 09:09:37.955015 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:37.951183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:42819 (local address 127.0.0.1:40693)
0620 09:09:37.951589 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:37.951605 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:37.951642 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:09:37.952258 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:37.952275 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:37.952556 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:37.952853 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:37.953730 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:37.953735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:37.953746 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:37.953755 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:37.953807 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:37.953825 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:37.953936 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:37.954056 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 09:09:37.954233 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":132,"threads_started":1}
I20260620 09:09:37.956498 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82: Bootstrap starting.
I20260620 09:09:37.957998 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:37.959254 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:37.960160 21940 log.cc:824] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:37.962224 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:37.960441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:53863 (local address 127.0.0.1:40693)
0620 09:09:37.960581 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:37.960586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:37.960798 (+ 212us) server_negotiation.cc:407] Connection header received
0620 09:09:37.960887 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:37.960893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:37.960935 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:37.961029 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:37.961786 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:37.961789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:37.961791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:37.961801 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:37.961824 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:37.961834 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:37.961905 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:37.962024 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 09:09:37.962058 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 09:09:37.962699 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82: No bootstrap required, opened a new log
I20260620 09:09:37.966348 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } }
I20260620 09:09:37.966616 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:37.966710 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed07cd205cd34a698177a2204c6b1f82, State: Initialized, Role: FOLLOWER
I20260620 09:09:37.967139 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } }
I20260620 09:09:37.967294 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:37.967381 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:37.967483 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:37.968477 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } }
I20260620 09:09:37.968858 21940 leader_election.cc:302] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed07cd205cd34a698177a2204c6b1f82; no voters:
I20260620 09:09:37.969267 21940 leader_election.cc:288] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:37.969408 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:37.969632 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [term 1 LEADER]: Becoming Leader. State: Replica: ed07cd205cd34a698177a2204c6b1f82, State: Running, Role: LEADER
I20260620 09:09:37.970053 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } }
I20260620 09:09:37.970610 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:37.971753 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed07cd205cd34a698177a2204c6b1f82. Latest consensus state: current_term: 1 leader_uuid: "ed07cd205cd34a698177a2204c6b1f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } } }
I20260620 09:09:37.972132 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:37.972038 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed07cd205cd34a698177a2204c6b1f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed07cd205cd34a698177a2204c6b1f82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40693 } } }
I20260620 09:09:37.972477 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:37.972666 21961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:37.975499 21961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:37.981467 21961 catalog_manager.cc:1347] Generated new cluster ID: 6b86e416c1b94288bb3b73a1d26b3b5d
I20260620 09:09:37.981539 21961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:37.990614 21961 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:37.991015 21961 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:37.997855 21961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed07cd205cd34a698177a2204c6b1f82: Generated new TSK 0
I20260620 09:09:37.998706 21961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:38.004984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21872
2026-06-20T09:09:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:38 dist-test-slave-92t4 krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:38 dist-test-slave-92t4 krb5kdc[21971](info): set up 2 sockets
Jun 20 09:09:38 dist-test-slave-92t4 krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260620 09:09:39.041893 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:39.168993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:39.283066 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:39.283412 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:39.283499 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:39.287721 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:39.287835 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:39.287874 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:39.287914 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:39.287949 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:39.293484 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:39.294699 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:39.295833 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:39.304533 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:39.304524 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:39.304652 21984 server_base.cc:1034] running on GCE node
W20260620 09:09:39.304504 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:39.305495 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:39.306655 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:39.307850 21984 hybrid_clock.cc:648] HybridClock initialized: now 1781946579307836 us; error 30 us; skew 500 ppm
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:39.310659 21984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:39.311591 21984 webserver.cc:460] Webserver started at http://127.0.98.62:41797/ using document root <none> and password file <none>
I20260620 09:09:39.312122 21984 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:39.312188 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:39.312366 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:39.314489 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9"
format_stamp: "Formatted at 2026-06-20 09:09:39 on dist-test-slave-92t4"
server_key: "5098a32159d160db5b8d51cc2f52b9db"
server_key_iv: "3e27ab1b07677e4d73a48ea8454b77b7"
server_key_version: "encryptionkey@0"
I20260620 09:09:39.315021 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9"
format_stamp: "Formatted at 2026-06-20 09:09:39 on dist-test-slave-92t4"
server_key: "5098a32159d160db5b8d51cc2f52b9db"
server_key_iv: "3e27ab1b07677e4d73a48ea8454b77b7"
server_key_version: "encryptionkey@0"
I20260620 09:09:39.318902 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 09:09:39.321559 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:39.322862 21984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:39.323001 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9"
format_stamp: "Formatted at 2026-06-20 09:09:39 on dist-test-slave-92t4"
server_key: "5098a32159d160db5b8d51cc2f52b9db"
server_key_iv: "3e27ab1b07677e4d73a48ea8454b77b7"
server_key_version: "encryptionkey@0"
I20260620 09:09:39.323117 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:39.348266 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:39.351431 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:39.351671 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:39.358999 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40937
I20260620 09:09:39.359023 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40937 every 8 connection(s)
I20260620 09:09:39.360031 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:39.362893 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:39.366274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 21984
I20260620 09:09:39.366386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:39.366690 392 external_mini_cluster.cc:1428] Setting key 7ab2890b73fb4af171a77be6057893f1
I20260620 09:09:39.369197 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: Bootstrap starting.
I20260620 09:09:39.371871 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:39.372733 22052 log.cc:824] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:39.375209 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: No bootstrap required, opened a new log
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:39.379110 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } }
I20260620 09:09:39.379374 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:39.379453 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9699cc7ad24d1ca44e2d0c52c5e9e9, State: Initialized, Role: FOLLOWER
I20260620 09:09:39.380016 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } }
I20260620 09:09:39.380190 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:39.380278 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:39.380381 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:39.381613 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } }
I20260620 09:09:39.381982 22052 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9699cc7ad24d1ca44e2d0c52c5e9e9; no voters:
I20260620 09:09:39.382294 22052 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:39.382408 22057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:39.382475 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:39.368324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42801 (local address 127.0.98.62:40937)
0620 09:09:39.368757 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:39.368781 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:39.368819 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:09:39.369569 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:39.369589 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:39.369955 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:39.370252 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:39.371245 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:39.372365 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:39.373527 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:39.374051 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:39.374176 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:39.377161 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:39.377167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:39.377180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:39.377210 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:39.378979 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:39.379537 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:39.379542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:39.379547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:39.379656 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:39.380077 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:39.380079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:39.380080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:39.380642 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:39.380857 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:39.381246 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 09:09:39.381502 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":147,"threads_started":1}
I20260620 09:09:39.382666 22057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9699cc7ad24d1ca44e2d0c52c5e9e9, State: Running, Role: LEADER
I20260620 09:09:39.383015 22057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } }
I20260620 09:09:39.383590 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:39.386111 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:39.385747 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } } }
I20260620 09:09:39.386227 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:39.385759 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9699cc7ad24d1ca44e2d0c52c5e9e9. Latest consensus state: current_term: 1 leader_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9699cc7ad24d1ca44e2d0c52c5e9e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40937 } } }
I20260620 09:09:39.386336 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9 [sys.catalog]: This master's current role is: LEADER
W20260620 09:09:39.389070 22071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:39.389146 22071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:39.389204 22073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:39.390089 22073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:39.396613 22073 catalog_manager.cc:1347] Generated new cluster ID: c7924a04ae0949928fc6eee488c1f792
I20260620 09:09:39.396731 22073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:39.397964 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:39.388349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44049 (local address 127.0.98.62:40937)
0620 09:09:39.388464 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:39.388468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:39.389355 (+ 887us) server_negotiation.cc:407] Connection header received
0620 09:09:39.389543 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:39.389546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:39.389585 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:39.389668 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:39.390417 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:39.390957 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:39.391807 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:39.392095 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:39.392204 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:39.394053 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:39.394057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:39.394060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:39.394084 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:39.395871 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:39.396478 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:39.396482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:39.396484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:39.396572 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:39.396946 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:39.396949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:39.396951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:39.397209 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:39.397356 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:39.397716 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 09:09:39.397809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260620 09:09:39.402853 22073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:39.404103 22073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:39.414029 22073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9699cc7ad24d1ca44e2d0c52c5e9e9: Generated new TSK 0
I20260620 09:09:39.415037 22073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:39.426631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 21984
W20260620 09:09:39.431614 21969 connection.cc:535] client connection to 127.0.98.62:40937 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40937): Connection reset by peer (error 104)
2026-06-20T09:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[22083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[22083](info): set up 2 sockets
Jun 20 09:09:39 dist-test-slave-92t4 krb5kdc[22083](info): commencing operation
krb5kdc: starting...
W20260620 09:09:40.465251 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:40 dist-test-slave-92t4 krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:40.590238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:40.704296 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:40.704618 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:40.704715 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:40.708981 22096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:40.709072 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:40.709093 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:40.709156 22096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:40.709193 22096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:40.714593 22096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:40.715752 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:40.716838 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:40.725455 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:40.725509 22096 server_base.cc:1034] running on GCE node
W20260620 09:09:40.725442 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:40.725458 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:40.726356 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:40.727563 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:40.728737 22096 hybrid_clock.cc:648] HybridClock initialized: now 1781946580728723 us; error 33 us; skew 500 ppm
Jun 20 09:09:40 dist-test-slave-92t4 krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:40.731992 22096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:40.733125 22096 webserver.cc:460] Webserver started at http://127.0.98.62:33639/ using document root <none> and password file <none>
I20260620 09:09:40.733738 22096 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:40.733804 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:40.734031 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:40.736217 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "8c2a9beebace405184d823fe84161f12"
format_stamp: "Formatted at 2026-06-20 09:09:40 on dist-test-slave-92t4"
server_key: "6b9dbf6c45e6b014832669221410e90c"
server_key_iv: "bc509ed3168358ded75d13b427fb1447"
server_key_version: "encryptionkey@0"
I20260620 09:09:40.736820 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c2a9beebace405184d823fe84161f12"
format_stamp: "Formatted at 2026-06-20 09:09:40 on dist-test-slave-92t4"
server_key: "6b9dbf6c45e6b014832669221410e90c"
server_key_iv: "bc509ed3168358ded75d13b427fb1447"
server_key_version: "encryptionkey@0"
I20260620 09:09:40.740705 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 09:09:40.743276 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:40.744592 22096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:40.744755 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "8c2a9beebace405184d823fe84161f12"
format_stamp: "Formatted at 2026-06-20 09:09:40 on dist-test-slave-92t4"
server_key: "6b9dbf6c45e6b014832669221410e90c"
server_key_iv: "bc509ed3168358ded75d13b427fb1447"
server_key_version: "encryptionkey@0"
I20260620 09:09:40.744884 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:40.758446 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:40.761991 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:40.762223 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:40.770159 22096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45239
I20260620 09:09:40.770161 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45239 every 8 connection(s)
I20260620 09:09:40.771271 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:40.774431 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:40.777323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22096
I20260620 09:09:40.777487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:40.777778 392 external_mini_cluster.cc:1428] Setting key 41b795466fcc9a3ea90c43083e3ac326
I20260620 09:09:40.780843 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12: Bootstrap starting.
I20260620 09:09:40.783656 22164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:40.784451 22164 log.cc:824] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:40.786724 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12: No bootstrap required, opened a new log
I20260620 09:09:40.788364 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:40.779356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39913 (local address 127.0.98.62:45239)
0620 09:09:40.779888 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:40.779909 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:40.779955 (+ 46us) server_negotiation.cc:407] Connection header received
0620 09:09:40.780779 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:40.780801 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:40.781285 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:40.781671 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:40.782121 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:40.783600 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:40.784593 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:40.785279 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:40.785460 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:40.786342 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:40.786349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:40.786365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:40.786887 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:40.786953 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:40.786982 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:40.787125 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:40.787295 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 09:09:40.787426 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":172,"threads_started":1}
I20260620 09:09:40.791158 22164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } }
I20260620 09:09:40.791512 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:40.791491 22164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:40.791663 22164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2a9beebace405184d823fe84161f12, State: Initialized, Role: FOLLOWER
I20260620 09:09:40.792204 22164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } }
I20260620 09:09:40.792342 22164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:40.792438 22164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:40.792536 22164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:40.793684 22164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } }
I20260620 09:09:40.794070 22164 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c2a9beebace405184d823fe84161f12; no voters:
I20260620 09:09:40.794466 22164 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:40.794567 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:40.794868 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [term 1 LEADER]: Becoming Leader. State: Replica: 8c2a9beebace405184d823fe84161f12, State: Running, Role: LEADER
I20260620 09:09:40.795265 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } }
I20260620 09:09:40.795802 22164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:40.797065 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c2a9beebace405184d823fe84161f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } } }
I20260620 09:09:40.797210 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:40.797441 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2a9beebace405184d823fe84161f12. Latest consensus state: current_term: 1 leader_uuid: "8c2a9beebace405184d823fe84161f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c2a9beebace405184d823fe84161f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45239 } } }
I20260620 09:09:40.797529 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:40.797911 22184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:40.798839 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:40.793822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59015 (local address 127.0.98.62:45239)
0620 09:09:40.793936 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:40.793941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:40.794121 (+ 180us) server_negotiation.cc:407] Connection header received
0620 09:09:40.794224 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:40.794227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:40.794275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:40.794356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:40.794706 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:40.795435 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:40.796596 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:40.796999 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:40.797563 (+ 564us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:40.798303 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:40.798310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:40.798316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:40.798333 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:40.798361 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:40.798374 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:40.798470 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:40.798659 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 09:09:40.798696 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:09:40.801445 22184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:40.807503 22184 catalog_manager.cc:1347] Generated new cluster ID: 5ee3d2af9f0641dca78e09ea01576b3e
I20260620 09:09:40.807636 22184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:40.843454 22184 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:40.844434 22184 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:40.848789 22184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2a9beebace405184d823fe84161f12: Generated new TSK 0
I20260620 09:09:40.849679 22184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:40.857095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22096
W20260620 09:09:40.862659 22081 connection.cc:535] client connection to 127.0.98.62:45239 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45239): Connection reset by peer (error 104)
2026-06-20T09:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:40 dist-test-slave-92t4 krb5kdc[22195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:40 dist-test-slave-92t4 krb5kdc[22195](info): set up 2 sockets
Jun 20 09:09:40 dist-test-slave-92t4 krb5kdc[22195](info): commencing operation
krb5kdc: starting...
W20260620 09:09:41.893519 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:41 dist-test-slave-92t4 krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:42.020098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:42.139451 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:42.139787 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:42.139887 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:42.144202 22208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:42.144300 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:42.144340 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:42.144379 22208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:42.144414 22208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:42.150113 22208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:42.151253 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:42.152316 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:42.161135 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:42.161128 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:42.161128 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:42.161415 22208 server_base.cc:1034] running on GCE node
I20260620 09:09:42.162046 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:42.163429 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:42.164708 22208 hybrid_clock.cc:648] HybridClock initialized: now 1781946582164696 us; error 37 us; skew 500 ppm
Jun 20 09:09:42 dist-test-slave-92t4 krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:42.171227 22208 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:42.172231 22208 webserver.cc:460] Webserver started at http://127.0.98.62:36019/ using document root <none> and password file <none>
I20260620 09:09:42.172791 22208 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:42.172865 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:42.173098 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:42.175220 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "23c55afb7c1e4644aba4df59dae4bb6e"
format_stamp: "Formatted at 2026-06-20 09:09:42 on dist-test-slave-92t4"
server_key: "c1643b004a7e1e0fd9d209eaf7562f56"
server_key_iv: "8ac9f21b1625f3edf53d48395c0c71b3"
server_key_version: "encryptionkey@0"
I20260620 09:09:42.175761 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "23c55afb7c1e4644aba4df59dae4bb6e"
format_stamp: "Formatted at 2026-06-20 09:09:42 on dist-test-slave-92t4"
server_key: "c1643b004a7e1e0fd9d209eaf7562f56"
server_key_iv: "8ac9f21b1625f3edf53d48395c0c71b3"
server_key_version: "encryptionkey@0"
I20260620 09:09:42.179646 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:42.182158 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:42.183450 22208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:42.183580 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "23c55afb7c1e4644aba4df59dae4bb6e"
format_stamp: "Formatted at 2026-06-20 09:09:42 on dist-test-slave-92t4"
server_key: "c1643b004a7e1e0fd9d209eaf7562f56"
server_key_iv: "8ac9f21b1625f3edf53d48395c0c71b3"
server_key_version: "encryptionkey@0"
I20260620 09:09:42.183720 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:42.188406 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:42.191498 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:42.191695 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:42.198681 22208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41245
I20260620 09:09:42.198704 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41245 every 8 connection(s)
I20260620 09:09:42.199617 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:42.202518 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:42.206600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22208
I20260620 09:09:42.206766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:42.207037 392 external_mini_cluster.cc:1428] Setting key eb4e112a60543425f3f823c0dd7c057c
I20260620 09:09:42.208519 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e: Bootstrap starting.
I20260620 09:09:42.211565 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:42.212505 22276 log.cc:824] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:42.213192 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:42.208456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52823 (local address 127.0.98.62:41245)
0620 09:09:42.208922 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:42.208937 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:42.208974 (+ 37us) server_negotiation.cc:407] Connection header received
0620 09:09:42.210057 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:42.210083 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:42.210483 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:42.210803 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:42.211782 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:42.211787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:42.211800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:42.211811 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:42.211866 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:42.211886 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:42.211974 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:42.212122 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 09:09:42.212312 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":169,"threads_started":1}
I20260620 09:09:42.214623 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e: No bootstrap required, opened a new log
I20260620 09:09:42.216054 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:42.219388 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } }
I20260620 09:09:42.219693 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:42.219766 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c55afb7c1e4644aba4df59dae4bb6e, State: Initialized, Role: FOLLOWER
I20260620 09:09:42.220232 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:42.218492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40521 (local address 127.0.98.62:41245)
0620 09:09:42.218627 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:42.218631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:42.218837 (+ 206us) server_negotiation.cc:407] Connection header received
0620 09:09:42.218973 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:42.218975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:42.219005 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:42.219068 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:42.219784 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:42.219788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:42.219790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:42.219803 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:42.219838 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:42.219848 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:42.219951 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:42.220027 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 09:09:42.220064 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 09:09:42.220294 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } }
I20260620 09:09:42.220474 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:42.220541 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:42.220639 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:42.221699 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } }
I20260620 09:09:42.222059 22276 leader_election.cc:302] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c55afb7c1e4644aba4df59dae4bb6e; no voters:
I20260620 09:09:42.222434 22276 leader_election.cc:288] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:42.222590 22286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:42.222864 22286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [term 1 LEADER]: Becoming Leader. State: Replica: 23c55afb7c1e4644aba4df59dae4bb6e, State: Running, Role: LEADER
I20260620 09:09:42.223237 22286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } }
I20260620 09:09:42.223803 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:42.225097 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c55afb7c1e4644aba4df59dae4bb6e. Latest consensus state: current_term: 1 leader_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } } }
I20260620 09:09:42.224953 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c55afb7c1e4644aba4df59dae4bb6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41245 } } }
I20260620 09:09:42.225209 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:42.225322 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:42.225804 22297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:42.229437 22297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:42.235023 22297 catalog_manager.cc:1347] Generated new cluster ID: 4a51cd29e23e4b44bdacb9bd7e59325e
I20260620 09:09:42.235210 22297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:42.264789 22297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:42.265317 22297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:42.271934 22297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c55afb7c1e4644aba4df59dae4bb6e: Generated new TSK 0
I20260620 09:09:42.272795 22297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:42.281906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22208
2026-06-20T09:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:42 dist-test-slave-92t4 krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:42 dist-test-slave-92t4 krb5kdc[22307](info): set up 2 sockets
Jun 20 09:09:42 dist-test-slave-92t4 krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260620 09:09:43.325423 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:43.454572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:43.566390 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:43.566666 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:43.566710 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:43.570762 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:43.570837 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:43.570856 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:43.570873 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:43.570888 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:43.576454 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:43.577629 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:43.578650 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:43.587402 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:43.587399 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:43.587486 22320 server_base.cc:1034] running on GCE node
W20260620 09:09:43.587399 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:43.588204 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:43.589322 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:43.590476 22320 hybrid_clock.cc:648] HybridClock initialized: now 1781946583590453 us; error 36 us; skew 500 ppm
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:43.593209 22320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:43.594138 22320 webserver.cc:460] Webserver started at http://127.0.98.62:39699/ using document root <none> and password file <none>
I20260620 09:09:43.594699 22320 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:43.594758 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:43.594998 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:43.597029 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "05778e63751b43fc97508a189641eb2d"
format_stamp: "Formatted at 2026-06-20 09:09:43 on dist-test-slave-92t4"
server_key: "f606b6bcb888b4ecf47eab91d14cfea9"
server_key_iv: "4d41ef49cdbf74967ef43678332deb3e"
server_key_version: "encryptionkey@0"
I20260620 09:09:43.597676 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "05778e63751b43fc97508a189641eb2d"
format_stamp: "Formatted at 2026-06-20 09:09:43 on dist-test-slave-92t4"
server_key: "f606b6bcb888b4ecf47eab91d14cfea9"
server_key_iv: "4d41ef49cdbf74967ef43678332deb3e"
server_key_version: "encryptionkey@0"
I20260620 09:09:43.601349 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:09:43.603713 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:43.604940 22320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:43.605118 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "05778e63751b43fc97508a189641eb2d"
format_stamp: "Formatted at 2026-06-20 09:09:43 on dist-test-slave-92t4"
server_key: "f606b6bcb888b4ecf47eab91d14cfea9"
server_key_iv: "4d41ef49cdbf74967ef43678332deb3e"
server_key_version: "encryptionkey@0"
I20260620 09:09:43.605242 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:43.620914 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:43.624373 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:43.624629 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:43.631786 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39899
I20260620 09:09:43.631811 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39899 every 8 connection(s)
I20260620 09:09:43.632812 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:43.635627 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:43.641253 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d: Bootstrap starting.
I20260620 09:09:43.641405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22320
I20260620 09:09:43.641543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:43.641845 392 external_mini_cluster.cc:1428] Setting key dc2c9c9692a29ec6de5481bbfb66d483
I20260620 09:09:43.643939 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:43.644686 22388 log.cc:824] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:43.646790 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d: No bootstrap required, opened a new log
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:43.651521 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } }
I20260620 09:09:43.651808 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:43.651897 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05778e63751b43fc97508a189641eb2d, State: Initialized, Role: FOLLOWER
I20260620 09:09:43.652426 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } }
I20260620 09:09:43.652590 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:43.652670 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:43.652778 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:43.654187 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } }
I20260620 09:09:43.654597 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05778e63751b43fc97508a189641eb2d; no voters:
I20260620 09:09:43.654994 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:43.655112 22393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:43.655382 22393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [term 1 LEADER]: Becoming Leader. State: Replica: 05778e63751b43fc97508a189641eb2d, State: Running, Role: LEADER
I20260620 09:09:43.655773 22393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } }
I20260620 09:09:43.656327 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:43.657166 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05778e63751b43fc97508a189641eb2d. Latest consensus state: current_term: 1 leader_uuid: "05778e63751b43fc97508a189641eb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } } }
I20260620 09:09:43.657334 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:43.657737 22402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:43.658159 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:43.643234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38059 (local address 127.0.98.62:39899)
0620 09:09:43.643674 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:43.643690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:43.643726 (+ 36us) server_negotiation.cc:407] Connection header received
0620 09:09:43.644416 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:43.644435 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:43.644831 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:43.645241 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:43.646038 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:43.647077 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:43.648227 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:43.648701 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:43.648816 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:43.651947 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:43.651975 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:43.651998 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:43.652045 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:43.654346 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:43.654932 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:43.654941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:43.654949 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:43.655104 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:43.655476 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:43.655479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:43.655481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:43.656045 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:43.656218 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:43.656690 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 09:09:43.657007 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260620 09:09:43.659340 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05778e63751b43fc97508a189641eb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05778e63751b43fc97508a189641eb2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39899 } } }
I20260620 09:09:43.659444 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:43.661697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:43.661934 22402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:43.667629 22402 catalog_manager.cc:1347] Generated new cluster ID: 0260e9245afd43778eb3835228b8fb50
I20260620 09:09:43.667697 22402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:43.674122 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:43.664248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37443 (local address 127.0.98.62:39899)
0620 09:09:43.664362 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:43.664367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:43.665427 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 09:09:43.665624 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:43.665629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:43.665683 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:43.665772 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:43.666605 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:43.667390 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:43.668582 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:43.668950 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:43.669064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:43.670716 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:43.670720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:43.670722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:43.670746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:43.672399 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:43.672876 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:43.672880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:43.672882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:43.672960 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:43.673287 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:43.673290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:43.673292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:43.673482 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:43.673593 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:43.673854 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 09:09:43.673962 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:09:43.683236 22402 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:43.684245 22402 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:43.691223 22402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05778e63751b43fc97508a189641eb2d: Generated new TSK 0
I20260620 09:09:43.691968 22402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:43.701161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22320
2026-06-20T09:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22419](info): set up 2 sockets
Jun 20 09:09:43 dist-test-slave-92t4 krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260620 09:09:44.741438 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:44 dist-test-slave-92t4 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:44.864596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:44.975871 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:44.976155 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:44.976199 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:44.980145 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:44.980202 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:44.980222 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:44.980238 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:44.980253 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:44.985564 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:44.986568 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:44.987551 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:44.998447 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:44.998508 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:44.998721 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:44.998775 22432 server_base.cc:1034] running on GCE node
I20260620 09:09:44.999899 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:45.001646 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:45.002887 22432 hybrid_clock.cc:648] HybridClock initialized: now 1781946585002820 us; error 88 us; skew 500 ppm
Jun 20 09:09:45 dist-test-slave-92t4 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:45.007066 22432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:45.008391 22432 webserver.cc:460] Webserver started at http://127.0.98.62:40679/ using document root <none> and password file <none>
I20260620 09:09:45.009207 22432 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:45.009305 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:45.009572 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:45.012439 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "14c588b64daf4394ab36aba88ba70b62"
format_stamp: "Formatted at 2026-06-20 09:09:45 on dist-test-slave-92t4"
server_key: "c660745eef5da3045c2adee6a3332ece"
server_key_iv: "a8371b4f18da5691cc908ebc90c3ef0a"
server_key_version: "encryptionkey@0"
I20260620 09:09:45.013252 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "14c588b64daf4394ab36aba88ba70b62"
format_stamp: "Formatted at 2026-06-20 09:09:45 on dist-test-slave-92t4"
server_key: "c660745eef5da3045c2adee6a3332ece"
server_key_iv: "a8371b4f18da5691cc908ebc90c3ef0a"
server_key_version: "encryptionkey@0"
I20260620 09:09:45.018862 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 09:09:45.022529 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:45.024156 22432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 09:09:45.024308 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "14c588b64daf4394ab36aba88ba70b62"
format_stamp: "Formatted at 2026-06-20 09:09:45 on dist-test-slave-92t4"
server_key: "c660745eef5da3045c2adee6a3332ece"
server_key_iv: "a8371b4f18da5691cc908ebc90c3ef0a"
server_key_version: "encryptionkey@0"
I20260620 09:09:45.024449 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:45.039390 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:45.042964 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:45.043182 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:45.050230 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43357
I20260620 09:09:45.050252 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43357 every 8 connection(s)
I20260620 09:09:45.051290 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:45.052078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22432
I20260620 09:09:45.052218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:45.052533 392 external_mini_cluster.cc:1428] Setting key ec4a5e74c577892e7600f4cc891904e4
I20260620 09:09:45.054847 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:45.061697 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62: Bootstrap starting.
I20260620 09:09:45.062323 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:45.054111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41355 (local address 127.0.98.62:43357)
0620 09:09:45.054579 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:45.054595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:45.054633 (+ 38us) server_negotiation.cc:407] Connection header received
0620 09:09:45.055388 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:45.055409 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:45.055816 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:45.056107 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:45.056528 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:45.057640 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:45.058918 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:45.059397 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:45.059529 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:45.060449 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:45.060454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:45.060468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:45.060893 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:45.060946 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:45.060969 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:45.061105 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:45.061247 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 09:09:45.061476 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":158,"threads_started":1}
I20260620 09:09:45.064251 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:45.065013 22500 log.cc:824] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:45.065208 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:45.067322 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62: No bootstrap required, opened a new log
I20260620 09:09:45.071581 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:45.067328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56531 (local address 127.0.98.62:43357)
0620 09:09:45.067458 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:45.067462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:45.067663 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:09:45.067809 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:45.067811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:45.067847 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:45.067909 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:45.068208 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:45.068700 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:45.069843 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:45.070061 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:45.070151 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:45.071045 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:45.071048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:45.071050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:45.071069 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:45.071095 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:45.071112 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:45.071181 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:45.071332 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 09:09:45.071383 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:09:45.072065 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } }
I20260620 09:09:45.072299 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:45.072356 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c588b64daf4394ab36aba88ba70b62, State: Initialized, Role: FOLLOWER
I20260620 09:09:45.072829 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } }
I20260620 09:09:45.072956 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:45.073067 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:45.073189 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:45.074113 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } }
I20260620 09:09:45.074462 22500 leader_election.cc:302] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c588b64daf4394ab36aba88ba70b62; no voters:
I20260620 09:09:45.074764 22500 leader_election.cc:288] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:45.074879 22510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:45.075139 22510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [term 1 LEADER]: Becoming Leader. State: Replica: 14c588b64daf4394ab36aba88ba70b62, State: Running, Role: LEADER
I20260620 09:09:45.075472 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } }
I20260620 09:09:45.076097 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:45.077133 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c588b64daf4394ab36aba88ba70b62. Latest consensus state: current_term: 1 leader_uuid: "14c588b64daf4394ab36aba88ba70b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } } }
I20260620 09:09:45.077262 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:45.077646 22519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:45.078496 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14c588b64daf4394ab36aba88ba70b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c588b64daf4394ab36aba88ba70b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43357 } } }
I20260620 09:09:45.078593 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:45.080447 22519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:45.086463 22519 catalog_manager.cc:1347] Generated new cluster ID: d3ac339c8d694d83a5fc3e45ed456dd6
I20260620 09:09:45.086566 22519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:45.098874 22519 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:45.100247 22519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:45.109606 22519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14c588b64daf4394ab36aba88ba70b62: Generated new TSK 0
I20260620 09:09:45.110482 22519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:45.114748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22432
2026-06-20T09:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:45 dist-test-slave-92t4 krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:45 dist-test-slave-92t4 krb5kdc[22531](info): set up 2 sockets
Jun 20 09:09:45 dist-test-slave-92t4 krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260620 09:09:46.165427 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:46 dist-test-slave-92t4 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:46.288995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:46.401960 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:46.402245 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:46.402288 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:46.406512 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:46.406580 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:46.406601 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:46.406618 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:46.406634 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:46.412181 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:46.413522 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:46.414626 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:46.423457 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:46.423467 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:46.423540 22544 server_base.cc:1034] running on GCE node
W20260620 09:09:46.423452 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:46.424340 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:46.425562 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:46.426750 22544 hybrid_clock.cc:648] HybridClock initialized: now 1781946586426739 us; error 31 us; skew 500 ppm
Jun 20 09:09:46 dist-test-slave-92t4 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:46.429507 22544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:46.430439 22544 webserver.cc:460] Webserver started at http://127.0.98.62:45717/ using document root <none> and password file <none>
I20260620 09:09:46.431011 22544 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:46.431102 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:46.431340 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:46.433506 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "3119de7eeb2d4fcb9aa520fe66e2182a"
format_stamp: "Formatted at 2026-06-20 09:09:46 on dist-test-slave-92t4"
server_key: "20ca68fa0956cadf3e81d6ef7b02fe0f"
server_key_iv: "13bc1b2af500a6b35a3467b51ee2f4e2"
server_key_version: "encryptionkey@0"
I20260620 09:09:46.434095 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "3119de7eeb2d4fcb9aa520fe66e2182a"
format_stamp: "Formatted at 2026-06-20 09:09:46 on dist-test-slave-92t4"
server_key: "20ca68fa0956cadf3e81d6ef7b02fe0f"
server_key_iv: "13bc1b2af500a6b35a3467b51ee2f4e2"
server_key_version: "encryptionkey@0"
I20260620 09:09:46.437932 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 09:09:46.440335 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:46.441653 22544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 09:09:46.441797 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "3119de7eeb2d4fcb9aa520fe66e2182a"
format_stamp: "Formatted at 2026-06-20 09:09:46 on dist-test-slave-92t4"
server_key: "20ca68fa0956cadf3e81d6ef7b02fe0f"
server_key_iv: "13bc1b2af500a6b35a3467b51ee2f4e2"
server_key_version: "encryptionkey@0"
I20260620 09:09:46.441918 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:46.446486 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:46.449292 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:46.449487 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:46.457144 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41539
I20260620 09:09:46.457177 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41539 every 8 connection(s)
I20260620 09:09:46.458177 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:46.461143 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:46.465703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22544
I20260620 09:09:46.465833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:46.466145 392 external_mini_cluster.cc:1428] Setting key 0ae042d0237ce0f514abfcc55128d425
I20260620 09:09:46.467331 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a: Bootstrap starting.
I20260620 09:09:46.469798 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:46.470561 22612 log.cc:824] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:46.471444 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:46.467609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35543 (local address 127.0.98.62:41539)
0620 09:09:46.468050 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:46.468065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:46.468100 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:09:46.468753 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:46.468770 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:46.469121 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:46.469477 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:46.470259 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:46.470264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:46.470277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:46.470288 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:46.470340 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:46.470358 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:46.470474 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:46.470575 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 09:09:46.470733 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260620 09:09:46.472625 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a: No bootstrap required, opened a new log
I20260620 09:09:46.474368 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:46.476758 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } }
I20260620 09:09:46.477101 22612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 09:09:46.477169 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3119de7eeb2d4fcb9aa520fe66e2182a, State: Initialized, Role: FOLLOWER
I20260620 09:09:46.477705 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } }
I20260620 09:09:46.477892 22612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 09:09:46.477984 22612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 09:09:46.478027 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:46.476344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55643 (local address 127.0.98.62:41539)
0620 09:09:46.476474 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:46.476479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:46.476641 (+ 162us) server_negotiation.cc:407] Connection header received
0620 09:09:46.476751 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:46.476753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:46.476780 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:46.476843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:46.477590 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:46.477593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:46.477595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 09:09:46.477604 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:46.477628 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 09:09:46.477638 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:46.477713 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:46.477817 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 09:09:46.477868 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:09:46.478092 22612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:46.479218 22612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } }
I20260620 09:09:46.479568 22612 leader_election.cc:302] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3119de7eeb2d4fcb9aa520fe66e2182a; no voters:
I20260620 09:09:46.479893 22612 leader_election.cc:288] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 09:09:46.479971 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:46.480197 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [term 1 LEADER]: Becoming Leader. State: Replica: 3119de7eeb2d4fcb9aa520fe66e2182a, State: Running, Role: LEADER
I20260620 09:09:46.480541 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } }
I20260620 09:09:46.481194 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:46.482328 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3119de7eeb2d4fcb9aa520fe66e2182a. Latest consensus state: current_term: 1 leader_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } } }
I20260620 09:09:46.482451 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:46.482771 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3119de7eeb2d4fcb9aa520fe66e2182a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41539 } } }
I20260620 09:09:46.482858 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:46.482839 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:46.486340 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:46.492030 22631 catalog_manager.cc:1347] Generated new cluster ID: fd80886145d14240ae6c1bec9e67422e
I20260620 09:09:46.492110 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:46.507141 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:46.507524 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:46.515183 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3119de7eeb2d4fcb9aa520fe66e2182a: Generated new TSK 0
I20260620 09:09:46.515910 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 09:09:46.528833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22544
2026-06-20T09:09:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1416 ms)
[----------] 18 tests from AuthTokenIssuingTest (25708 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:46 dist-test-slave-92t4 krb5kdc[22643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:46 dist-test-slave-92t4 krb5kdc[22643](info): set up 2 sockets
Jun 20 09:09:46 dist-test-slave-92t4 krb5kdc[22643](info): commencing operation
krb5kdc: starting...
W20260620 09:09:47.573572 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:47 dist-test-slave-92t4 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:47.696388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:47.809370 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:47.809746 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:47.809789 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:47.809875 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:47.814004 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:47.814123 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:47.814162 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:47.814201 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:47.814265 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:47.820222 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:47.821511 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:47.822643 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:47.831552 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:47.831557 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:47.831535 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:47.832031 22656 server_base.cc:1034] running on GCE node
I20260620 09:09:47.832564 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:47.833787 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:47.834980 22656 hybrid_clock.cc:648] HybridClock initialized: now 1781946587834963 us; error 29 us; skew 500 ppm
Jun 20 09:09:47 dist-test-slave-92t4 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:47.841107 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:47.842092 22656 webserver.cc:460] Webserver started at http://127.0.98.62:42445/ using document root <none> and password file <none>
I20260620 09:09:47.842702 22656 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:47.842795 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:47.843030 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:47.845252 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "536b48b681914317ab4ac5104fe8c0fe"
format_stamp: "Formatted at 2026-06-20 09:09:47 on dist-test-slave-92t4"
server_key: "aa6efa9dffea5e110708bff176fc5b4f"
server_key_iv: "114a9cfa62fb6b97363040a43587e570"
server_key_version: "encryptionkey@0"
I20260620 09:09:47.845834 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "536b48b681914317ab4ac5104fe8c0fe"
format_stamp: "Formatted at 2026-06-20 09:09:47 on dist-test-slave-92t4"
server_key: "aa6efa9dffea5e110708bff176fc5b4f"
server_key_iv: "114a9cfa62fb6b97363040a43587e570"
server_key_version: "encryptionkey@0"
I20260620 09:09:47.849663 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 09:09:47.852209 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:47.853729 22656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 09:09:47.853888 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "536b48b681914317ab4ac5104fe8c0fe"
format_stamp: "Formatted at 2026-06-20 09:09:47 on dist-test-slave-92t4"
server_key: "aa6efa9dffea5e110708bff176fc5b4f"
server_key_iv: "114a9cfa62fb6b97363040a43587e570"
server_key_version: "encryptionkey@0"
I20260620 09:09:47.854043 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:47.874464 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:47.877429 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:47.877655 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:47.884881 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45877
I20260620 09:09:47.885175 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45877 every 8 connection(s)
I20260620 09:09:47.885946 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:47.888801 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:47.892962 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:47.893508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22656
I20260620 09:09:47.893703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:47.893985 392 external_mini_cluster.cc:1428] Setting key 8044d0b7d5c0743b2d2295db5cd67165
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 09:09:47 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:47.911593 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.895740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.0.98.62:45877)
0620 09:09:47.896355 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:47.896371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:47.896418 (+ 47us) server_negotiation.cc:407] Connection header received
0620 09:09:47.897577 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:47.897586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:47.897847 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:47.898072 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:47.898910 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:47.900447 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:47.901525 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:47.902171 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:47.902395 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:47.906058 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:47.906064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:47.906075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:47.906095 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:47.908181 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:47.908670 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:47.908675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:47.908677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:47.908795 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:47.909281 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:47.909285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:47.909286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:47.909691 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:47.909849 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:47.910212 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 09:09:47.910489 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":487,"spinlock_wait_cycles":15360,"thread_start_us":185,"threads_started":1}
I20260620 09:09:47.911593 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.895739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45877 (local address 127.0.0.1:43764)
0620 09:09:47.896251 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:47.896304 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:47.897249 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:47.898146 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:47.898150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:47.898484 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:47.898743 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:47.898756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:47.900593 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:47.900596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:47.901405 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:47.901414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:47.902337 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:47.902340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:47.902513 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:47.903176 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:47.903192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:47.905889 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:47.908296 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:47.908307 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:47.908319 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:47.908552 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:47.908891 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:47.908895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:47.908898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:47.909160 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:47.909818 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:47.909825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:47.909971 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 09:09:47.910140 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 09:09:47.910540 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":157,"spinlock_wait_cycles":7936,"thread_start_us":153,"threads_started":1}
I20260620 09:09:47.914458 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:47.915298 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.915042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57814)
0620 09:09:47.915157 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:47.915232 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 09:09:47.915490 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
I20260620 09:09:47.916301 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.916107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57818)
0620 09:09:47.916206 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:47.916250 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 09:09:47.916430 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:09:47.920344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45953
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:09:47.965687 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } attempt: 1
I20260620 09:09:47.966610 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.966254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57828)
0620 09:09:47.966439 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:47.966516 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 09:09:47.967365 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:47.967144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57844)
0620 09:09:47.967250 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:47.967301 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 09:09:47.967506 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
W20260620 09:09:48.034930 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:48.035247 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:48.035271 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:48.035311 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 09:09:48.035692 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } attempt: 2
I20260620 09:09:48.036563 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.036221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57846)
0620 09:09:48.036400 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.036472 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 09:09:48.037341 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.037130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57850)
0620 09:09:48.037231 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.037279 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 09:09:48.037525 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45953: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45953: connect: Connection refused (error 111)
W20260620 09:09:48.040148 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:48.040215 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:48.040241 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:48.040295 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:48.040328 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:48.046041 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:48.047219 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:48.048342 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:48.057727 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:48.057727 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:48.057767 22731 server_base.cc:1034] running on GCE node
W20260620 09:09:48.057772 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:48.058630 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:48.059804 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:48.061002 22731 hybrid_clock.cc:648] HybridClock initialized: now 1781946588060982 us; error 31 us; skew 500 ppm
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:48.064033 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:09:48.065027 22731 webserver.cc:460] Webserver started at http://127.0.98.61:41539/ using document root <none> and password file <none>
I20260620 09:09:48.065698 22731 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:48.065800 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:48.066036 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:48.068149 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data/instance:
uuid: "7e6ec637324d452982cb81e98b20c38c"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "42fa2dceff1da6b830aa830ac236219c"
server_key_iv: "ad9690fdf2158d562f2323d5d863fb5c"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.068744 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal/instance:
uuid: "7e6ec637324d452982cb81e98b20c38c"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "42fa2dceff1da6b830aa830ac236219c"
server_key_iv: "ad9690fdf2158d562f2323d5d863fb5c"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.072814 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 09:09:48.075397 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:48.076828 22731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 09:09:48.076975 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "7e6ec637324d452982cb81e98b20c38c"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "42fa2dceff1da6b830aa830ac236219c"
server_key_iv: "ad9690fdf2158d562f2323d5d863fb5c"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.077164 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:48.094308 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:48.097680 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:48.097913 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:48.105397 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45953
I20260620 09:09:48.105716 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45953 every 8 connection(s)
I20260620 09:09:48.106451 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:09:48.106813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22731
I20260620 09:09:48.106937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-1/wal/instance
I20260620 09:09:48.107246 392 external_mini_cluster.cc:1428] Setting key 68d007e4d5378c921a80a920e81c0bb6
I20260620 09:09:48.109553 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:48.115286 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:48.128800 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } attempt: 3
I20260620 09:09:48.130719 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.117708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.62:45877)
0620 09:09:48.117883 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.117889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.118780 (+ 891us) server_negotiation.cc:407] Connection header received
0620 09:09:48.119754 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.119758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.119819 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.119935 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.121168 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.121712 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.122885 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.123131 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.123258 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.125889 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.125893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.125895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.125920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.128063 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.128605 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.128609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.128610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.128738 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.129209 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.129213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.129214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.129459 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.129567 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.130311 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.130469 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 09:09:48.132803 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.118055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45877 (local address 127.0.0.1:43766)
0620 09:09:48.118618 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.118668 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.119537 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.120118 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.120125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.120472 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.120984 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.120997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.121853 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.121857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.122750 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.122759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.123296 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.123300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.123478 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.124141 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.124159 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.125692 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.128186 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.128193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.128213 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.128493 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.128852 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.128856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.128858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.129107 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.129572 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.129581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.130014 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.130938 (+ 924us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.131511 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":63,"thread_start_us":110,"threads_started":1}
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:09:48.141151 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.129297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57852)
0620 09:09:48.129482 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.129501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.129628 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.130234 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.130238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.130540 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.130675 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.130685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.132740 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.132744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.133559 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.133571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.134304 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.134309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.134428 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.134953 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.134965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.137228 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.139395 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.139399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.139401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.139616 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.139932 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.139937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.139939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.140030 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.140549 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.140552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.140655 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.140784 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.140941 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 09:09:48.141289 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.129421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.98.61:45953)
0620 09:09:48.129762 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.129775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.129798 (+ 23us) server_negotiation.cc:407] Connection header received
0620 09:09:48.129841 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.129847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.130068 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.130214 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.131102 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.132560 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.133692 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.134114 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.134258 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.137364 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.137375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.137391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.137425 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.139263 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.139735 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.139742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.139746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.139831 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.140129 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.140132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.140134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.140433 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.140590 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.140889 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.141083 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260620 09:09:48.143755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37491
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:09:48.145524 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.146735 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.147482 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.147218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42668)
0620 09:09:48.147335 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.147399 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 09:09:48.148343 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.148076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42682)
0620 09:09:48.148220 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.148274 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 09:09:48.149122 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:09:48.155525 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.146198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.98.61:45953)
0620 09:09:48.146335 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.146341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.146359 (+ 18us) server_negotiation.cc:407] Connection header received
0620 09:09:48.146464 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.146468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.146514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.146598 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.147200 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.147982 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.149157 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.149608 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.149725 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.152212 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.152215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.152217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.152239 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.153722 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.154223 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.154225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.154226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.154295 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.154606 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.154608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.154609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.154773 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.154845 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.155198 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.155390 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":64}
I20260620 09:09:48.155615 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.146100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57858)
0620 09:09:48.146285 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.146299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.146379 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.146735 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.146740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.146934 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.147078 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.147088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.148102 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.148107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.148992 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.149009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.149862 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.149868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.149999 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.150479 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.150500 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.152038 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.153832 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.153837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.153841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.154122 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.154387 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.154393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.154398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.154510 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.154870 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.154876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.155002 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.155127 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.155320 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 09:09:48.156450 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.157270 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.156958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42694)
0620 09:09:48.157097 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.157188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:09:48.157526 22747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
I20260620 09:09:48.158429 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.158202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42710)
0620 09:09:48.158323 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.158372 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 09:09:48.158564 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
I20260620 09:09:48.180308 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 1
I20260620 09:09:48.181260 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.180879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42714)
0620 09:09:48.181076 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.181154 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 09:09:48.182119 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.181824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42726)
0620 09:09:48.181984 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.182049 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 09:09:48.182307 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
I20260620 09:09:48.207865 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 1
I20260620 09:09:48.208861 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.208479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42738)
0620 09:09:48.208674 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.208759 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 09:09:48.209651 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.209435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42740)
0620 09:09:48.209542 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.209595 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:09:48.209786 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
I20260620 09:09:48.257493 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 2
I20260620 09:09:48.258493 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.258132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42746)
0620 09:09:48.258320 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.258394 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 09:09:48.259305 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.259068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42754)
0620 09:09:48.259175 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.259237 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:09:48.259438 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
W20260620 09:09:48.263506 22805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:48.263832 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:48.263857 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:48.263899 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:48.268061 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:48.268119 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:48.268141 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:48.268157 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:48.268172 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:48.273777 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45877,127.0.98.61:45953,127.0.98.60:37491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:48.274866 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:48.275890 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:48.277954 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 2
I20260620 09:09:48.278832 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.278480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42764)
0620 09:09:48.278657 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.278737 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 09:09:48.279589 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.279355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42766)
0620 09:09:48.279468 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.279521 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:09:48.279752 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37491: connect: Connection refused (error 111)
W20260620 09:09:48.285018 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:48.285085 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:48.285159 22805 server_base.cc:1034] running on GCE node
W20260620 09:09:48.285102 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:48.286053 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:48.287204 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:48.288398 22805 hybrid_clock.cc:648] HybridClock initialized: now 1781946588288376 us; error 36 us; skew 500 ppm
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:48.291195 22805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:09:48.292164 22805 webserver.cc:460] Webserver started at http://127.0.98.60:44735/ using document root <none> and password file <none>
I20260620 09:09:48.292754 22805 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:48.292845 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:48.293139 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:48.295346 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data/instance:
uuid: "30d8e8dfbc774d34857bc045bf20d791"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "4af123782088b118388c4f5b48483a72"
server_key_iv: "afd7c104706b38f86c055fd3623dadc3"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.295953 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal/instance:
uuid: "30d8e8dfbc774d34857bc045bf20d791"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "4af123782088b118388c4f5b48483a72"
server_key_iv: "afd7c104706b38f86c055fd3623dadc3"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.299911 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 09:09:48.302558 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:48.303810 22805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:48.303954 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "30d8e8dfbc774d34857bc045bf20d791"
format_stamp: "Formatted at 2026-06-20 09:09:48 on dist-test-slave-92t4"
server_key: "4af123782088b118388c4f5b48483a72"
server_key_iv: "afd7c104706b38f86c055fd3623dadc3"
server_key_version: "encryptionkey@0"
I20260620 09:09:48.304077 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:48.330260 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:48.333604 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:48.333847 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:48.341267 22805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37491
I20260620 09:09:48.341589 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37491 every 8 connection(s)
I20260620 09:09:48.342319 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:09:48.345228 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:48.348944 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.352378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 22805
I20260620 09:09:48.352502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781946309842113-392-0/minicluster-data/master-2/wal/instance
I20260620 09:09:48.352742 392 external_mini_cluster.cc:1428] Setting key 60db09520aa29b3212a6657162621058
I20260620 09:09:48.358577 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 3
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:09:48.365475 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.350935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.0.98.62:45877)
0620 09:09:48.351087 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.351092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.352069 (+ 977us) server_negotiation.cc:407] Connection header received
0620 09:09:48.353227 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.353233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.353292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.353399 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.355001 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.355799 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.357090 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.357330 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.357436 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.360594 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.360597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.360618 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.360650 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.362441 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.363282 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.363288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.363291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.363388 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.363802 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.363806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.363808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.364054 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.364159 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.365139 (+ 980us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.365276 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 09:09:48.366880 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.353415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.60:37491)
0620 09:09:48.353801 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.353809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.353830 (+ 21us) server_negotiation.cc:407] Connection header received
0620 09:09:48.353959 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.353963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.354116 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.354252 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.355153 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.356249 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.357353 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.357769 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.357851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.361546 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.361552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.361565 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.361589 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.364157 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.364613 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.364619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.364620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.364718 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.365003 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.365008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.365009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.365324 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.365542 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.365774 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.365980 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260620 09:09:48.366883 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.351399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45877 (local address 127.0.0.1:43774)
0620 09:09:48.351898 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.351953 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.352950 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.353738 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.353749 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.354180 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.354842 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.354863 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.355912 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.355915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.356930 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.356940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.357473 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.357476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.357683 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.358354 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.358380 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.360451 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.362825 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.362834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.362846 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.363156 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.363505 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.363508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.363510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.363695 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.364164 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.364170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.364513 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.365100 (+ 587us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.366096 (+ 996us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":119,"spinlock_wait_cycles":15104,"thread_start_us":156,"threads_started":1}
I20260620 09:09:48.368342 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.368865 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.359025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42780)
0620 09:09:48.359141 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.359157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.359262 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.359638 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.359640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.359780 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.359853 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.359858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.360883 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.360886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.361728 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.361740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.362572 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.362576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.362639 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.363026 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.363037 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.364243 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.367327 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.367331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.367333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.367553 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.367880 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.367883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.367885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.367971 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.368374 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.368377 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.368472 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.368583 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.368711 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 09:09:48.369303 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.359077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.98.60:37491)
0620 09:09:48.359398 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.359405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.359434 (+ 29us) server_negotiation.cc:407] Connection header received
0620 09:09:48.359468 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.359471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.359509 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.359606 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.359963 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.360763 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.362071 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.362455 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.362550 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.364342 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.364345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.364348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.364369 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.367235 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.367676 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.367682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.367685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.367777 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.368076 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.368079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.368080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.368266 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.368365 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.368662 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.368787 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260620 09:09:48.370949 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } attempt: 3
I20260620 09:09:48.372840 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: Bootstrap starting.
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:48.376160 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:48.377215 22724 log.cc:824] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:48.378247 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.369099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.98.61:45953)
0620 09:09:48.369237 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.369242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.369282 (+ 40us) server_negotiation.cc:407] Connection header received
0620 09:09:48.369456 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.369460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.369515 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.369574 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.370132 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.370642 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.371854 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.372104 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.372216 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.374681 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.374685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.374687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.374721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.376325 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.376779 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.376785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.376790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.376884 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.377270 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.377275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.377277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.377510 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.377594 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.377956 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.378071 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 09:09:48.378727 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.368798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45953 (local address 127.0.0.1:57864)
0620 09:09:48.369222 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.369236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.369325 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.369719 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.369721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.369894 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.370004 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.370010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.370752 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.370758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.371732 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.371740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.372213 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.372216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.372315 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.373136 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.373149 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.374523 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.376458 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.376461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.376462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.376645 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.377003 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.377005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.377006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.377159 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.377666 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.377669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.377774 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.378449 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.378576 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260620 09:09:48.379642 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: No bootstrap required, opened a new log
I20260620 09:09:48.379945 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:48.380100 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.369841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.0.98.62:45877)
0620 09:09:48.369943 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.369949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.369960 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:09:48.369992 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.369995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.370029 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.370127 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.370878 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.371535 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.372623 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.372931 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.373107 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.375886 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.375890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.375893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.375919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.377690 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.378214 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.378218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.378219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.378535 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.379092 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.379095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.379097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.379309 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.379452 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.379836 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.379957 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:09:48.383970 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.384207 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536b48b681914317ab4ac5104fe8c0fe, State: Initialized, Role: FOLLOWER
I20260620 09:09:48.384621 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.385222 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.385464 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:09:48.386395 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.371308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42784)
0620 09:09:48.377222 (+ 5914us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.377240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.377316 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.377599 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.377603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.377779 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.377900 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.377906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.378940 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.378943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.379523 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.379531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.380754 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.380758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.380844 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.381270 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.381280 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.382605 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.384769 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.384773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.384776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.385065 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.385392 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.385395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.385397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.385487 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.385938 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.385941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.386032 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.386112 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.386248 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5814}
I20260620 09:09:48.387167 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:48.387393 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.371511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.0.98.60:37491)
0620 09:09:48.371610 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.371614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.377396 (+ 5782us) server_negotiation.cc:407] Connection header received
0620 09:09:48.377432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.377434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.377475 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.377998 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.378088 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.378811 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.379905 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.380637 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.380743 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.382760 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.382764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.382766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.382789 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.384412 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.385185 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.385190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.385193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.385284 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.385604 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.385607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.385609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.385824 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.387068 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.387142 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.387259 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 09:09:48.389990 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.380423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37491 (local address 127.0.0.1:42792)
0620 09:09:48.380867 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:48.380880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:48.381012 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:48.381462 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:48.381465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:48.381624 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:48.381726 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.381736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.382695 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.382701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.383519 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:48.383534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.384255 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.384258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:48.384354 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.384736 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:48.384746 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:48.386141 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:48.388573 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.388576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.388577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.388820 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.389116 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:48.389119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:48.389121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:48.389196 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:48.389554 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:48.389556 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:48.389636 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 09:09:48.389726 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 09:09:48.389845 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386}
I20260620 09:09:48.390477 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.380678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.60:37491)
0620 09:09:48.380792 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.380796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.380945 (+ 149us) server_negotiation.cc:407] Connection header received
0620 09:09:48.381287 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.381290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.381337 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.381419 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.381852 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.382569 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.383639 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.384033 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.384149 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.386295 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.386299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.386301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.386323 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.388459 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.388913 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.388916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.388918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.388985 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.389278 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.389281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.389283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.389444 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.389527 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.390235 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.390342 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:48.390847 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: Bootstrap starting.
I20260620 09:09:48.393508 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: Bootstrap starting.
I20260620 09:09:48.393700 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: Neither blocks nor log segments found. Creating new log.
Jun 20 09:09:48 dist-test-slave-92t4 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:09:48.394685 22799 log.cc:824] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:48.392117 22656 master_runner.cc:186] Error getting master registration for 127.0.98.61:45953: OK, instance_id { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" instance_seqno: 1781946588103304 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:09:48.396170 22656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:09:48.396517 22874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:48.396852 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: No bootstrap required, opened a new log
I20260620 09:09:48.397447 22874 log.cc:824] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:48.398622 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.386876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.0.98.61:45953)
0620 09:09:48.386979 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.386983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.387667 (+ 684us) server_negotiation.cc:407] Connection header received
0620 09:09:48.387812 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.387815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.387864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.387942 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.388818 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.389610 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.391132 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.391478 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.391554 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.394514 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.394518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.394521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.394545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.396356 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.397081 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.397093 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.397095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.397179 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.397504 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.397507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.397509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.397726 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.397903 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.398325 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.398441 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 09:09:48.399250 22892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:48.399323 22892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:48.401207 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.401448 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6ec637324d452982cb81e98b20c38c, State: Initialized, Role: FOLLOWER
I20260620 09:09:48.401640 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: No bootstrap required, opened a new log
I20260620 09:09:48.401810 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.401891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:48.402299 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.402513 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.403177 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:48.403277 22698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "536b48b681914317ab4ac5104fe8c0fe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 09:09:48.403515 22698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:09:48.403643 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:48.404856 22698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.405855 22698 leader_election.cc:288] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [CANDIDATE]: Term 1 election: Requested vote from peers 7e6ec637324d452982cb81e98b20c38c (127.0.98.61:45953), 30d8e8dfbc774d34857bc045bf20d791 (127.0.98.60:37491)
I20260620 09:09:48.406055 22874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.406356 22874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d8e8dfbc774d34857bc045bf20d791, State: Initialized, Role: FOLLOWER
I20260620 09:09:48.406685 22874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.406994 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "536b48b681914317ab4ac5104fe8c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e6ec637324d452982cb81e98b20c38c"
I20260620 09:09:48.407212 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:48.408043 22874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:48.408521 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536b48b681914317ab4ac5104fe8c0fe in term 1.
W20260620 09:09:48.408766 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:48.408836 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:48.409207 22848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "536b48b681914317ab4ac5104fe8c0fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30d8e8dfbc774d34857bc045bf20d791"
I20260620 09:09:48.409379 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:48.410301 22848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536b48b681914317ab4ac5104fe8c0fe in term 1.
I20260620 09:09:48.411418 22672 leader_election.cc:302] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d8e8dfbc774d34857bc045bf20d791, 536b48b681914317ab4ac5104fe8c0fe; no voters:
I20260620 09:09:48.411763 22881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:48.411959 22881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [term 1 LEADER]: Becoming Leader. State: Replica: 536b48b681914317ab4ac5104fe8c0fe, State: Running, Role: LEADER
I20260620 09:09:48.412317 22881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } }
I20260620 09:09:48.412685 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.412889 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.414268 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 536b48b681914317ab4ac5104fe8c0fe. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.414384 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:48.419364 22731 master_runner.cc:186] Error getting master registration for 127.0.98.60:37491: OK, instance_id { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" instance_seqno: 1781946588339142 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:09:48.423321 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:09:48.423543 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.412770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.0.98.62:45877)
0620 09:09:48.412883 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.412888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.414103 (+ 1215us) server_negotiation.cc:407] Connection header received
0620 09:09:48.414306 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.414309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.414359 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.414418 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.415506 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.416272 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.417486 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.417786 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.417864 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.419823 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.419828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.419831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.419859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.421664 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.422155 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.422159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.422161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.422254 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.422572 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.422575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.422577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.422781 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.422882 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.423245 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.423368 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 09:09:48.426301 22921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:48.427331 22921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 09:09:48.436440 22929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 09:09:48.437531 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.413472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.0.98.61:45953)
0620 09:09:48.413621 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.413627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.425096 (+ 11469us) server_negotiation.cc:407] Connection header received
0620 09:09:48.425289 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.425294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.425346 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.425397 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.429267 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.430019 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.431152 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.431541 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.431617 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.433396 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.433401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.433404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.433433 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.435441 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.435957 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.435962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.435966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.436057 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.436387 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.436390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.436392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.436662 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.436797 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.437199 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.437324 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 09:09:48.439273 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:48.429950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.0.98.60:37491)
0620 09:09:48.430079 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.430085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.430406 (+ 321us) server_negotiation.cc:407] Connection header received
0620 09:09:48.430566 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.430569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.430615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.430662 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.431589 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.432303 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.433513 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:48.433871 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:48.433957 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:48.435673 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:48.435677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:48.435681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:48.435705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:48.437480 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.437954 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.437959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.437960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.438058 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:48.438369 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:48.438372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:48.438373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:48.438571 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:48.438668 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:48.439031 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 09:09:48.439147 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 09:09:48.441143 22929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:48.446453 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [term 1 FOLLOWER]: Refusing update from remote peer 536b48b681914317ab4ac5104fe8c0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:48.446453 22848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [term 1 FOLLOWER]: Refusing update from remote peer 536b48b681914317ab4ac5104fe8c0fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:48.447168 22881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:48.447654 22881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:48.452466 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 536b48b681914317ab4ac5104fe8c0fe. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.452628 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.456477 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 536b48b681914317ab4ac5104fe8c0fe. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.456617 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.472890 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.473088 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:48.474068 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.474187 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.474741 22934 mvcc.cc:205] Tried to move back new op lower bound from 7298853226229403648 to 7298853226140786688. Current Snapshot: MvccSnapshot[applied={T|T < 7298853226229403648}]
I20260620 09:09:48.475452 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.475620 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:48.476660 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "536b48b681914317ab4ac5104fe8c0fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536b48b681914317ab4ac5104fe8c0fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45877 } } peers { permanent_uuid: "7e6ec637324d452982cb81e98b20c38c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45953 } } peers { permanent_uuid: "30d8e8dfbc774d34857bc045bf20d791" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37491 } } }
I20260620 09:09:48.476796 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:48.477906 22932 mvcc.cc:205] Tried to move back new op lower bound from 7298853226229403648 to 7298853226140786688. Current Snapshot: MvccSnapshot[applied={T|T < 7298853226229403648}]
I20260620 09:09:48.479720 22921 catalog_manager.cc:1347] Generated new cluster ID: 91fdc583a1004004a5f582be48786691
I20260620 09:09:48.479782 22921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:48.514168 22921 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:48.515502 22921 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:48.526798 22921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 536b48b681914317ab4ac5104fe8c0fe: Generated new TSK 0
I20260620 09:09:48.527751 22921 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:09:48.560602 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.557270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.98.62:45877)
0620 09:09:48.557408 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.557414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.559132 (+ 1718us) server_negotiation.cc:407] Connection header received
0620 09:09:48.559605 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.559609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.559662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.559717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.560405 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43804: BlockingRecv error: recv got EOF from 127.0.0.1:43804 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 09:09:48.562569 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.558084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.98.61:45953)
0620 09:09:48.558204 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.558210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.560997 (+ 2787us) server_negotiation.cc:407] Connection header received
0620 09:09:48.561795 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.561799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.561849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.561900 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.562433 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57880: BlockingRecv error: recv got EOF from 127.0.0.1:57880 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 09:09:48.563587 22878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.558924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.0.98.60:37491)
0620 09:09:48.559040 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.559044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.561710 (+ 2666us) server_negotiation.cc:407] Connection header received
0620 09:09:48.562897 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.562900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.562949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.562999 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.563456 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42812: BlockingRecv error: recv got EOF from 127.0.0.1:42812 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 09:09:48.568053 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.566011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.0.98.61:45953)
0620 09:09:48.566111 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.566115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.567157 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 09:09:48.567365 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.567368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.567438 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.567488 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.567955 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57892: BlockingRecv error: recv got EOF from 127.0.0.1:57892 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 09:09:48.572955 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.571406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.0.98.61:45953)
0620 09:09:48.571524 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.571529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.572057 (+ 528us) server_negotiation.cc:407] Connection header received
0620 09:09:48.572239 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.572242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.572291 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.572341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.572844 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57896: BlockingRecv error: recv got EOF from 127.0.0.1:57896 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 09:09:48.578668 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.577070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.0.98.61:45953)
0620 09:09:48.577183 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.577188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.577731 (+ 543us) server_negotiation.cc:407] Connection header received
0620 09:09:48.577924 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.577931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.577984 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.578038 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.578553 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57904: BlockingRecv error: recv got EOF from 127.0.0.1:57904 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 09:09:48.586552 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.584799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.98.61:45953)
0620 09:09:48.584943 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.584948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.585597 (+ 649us) server_negotiation.cc:407] Connection header received
0620 09:09:48.585792 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.585795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.585850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.585900 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.586421 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57908: BlockingRecv error: recv got EOF from 127.0.0.1:57908 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 09:09:48.599020 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.597219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.0.98.61:45953)
0620 09:09:48.597355 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.597360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.597993 (+ 633us) server_negotiation.cc:407] Connection header received
0620 09:09:48.598219 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.598223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.598280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.598336 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.598902 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57912: BlockingRecv error: recv got EOF from 127.0.0.1:57912 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 09:09:48.619614 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.617809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.98.61:45953)
0620 09:09:48.617944 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.617949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.618578 (+ 629us) server_negotiation.cc:407] Connection header received
0620 09:09:48.618808 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.618812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.618868 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.618921 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.619492 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57918: BlockingRecv error: recv got EOF from 127.0.0.1:57918 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 09:09:48.655694 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.654421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.98.61:45953)
0620 09:09:48.654580 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.654585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.654788 (+ 203us) server_negotiation.cc:407] Connection header received
0620 09:09:48.654900 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.654903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.654974 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.655075 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.655563 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57922: BlockingRecv error: recv got EOF from 127.0.0.1:57922 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 09:09:48.723404 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.722077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.0.98.61:45953)
0620 09:09:48.722244 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.722249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.722451 (+ 202us) server_negotiation.cc:407] Connection header received
0620 09:09:48.722596 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.722598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.722650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.722736 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.723271 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57928: BlockingRecv error: recv got EOF from 127.0.0.1:57928 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 09:09:48.855033 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:48.853751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.0.98.61:45953)
0620 09:09:48.853896 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:48.853901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:48.854102 (+ 201us) server_negotiation.cc:407] Connection header received
0620 09:09:48.854287 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:48.854290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:48.854339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:48.854432 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:48.854899 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57944: BlockingRecv error: recv got EOF from 127.0.0.1:57944 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 09:09:49.114509 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:49.113313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.98.61:45953)
0620 09:09:49.113454 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.113458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.113597 (+ 139us) server_negotiation.cc:407] Connection header received
0620 09:09:49.113748 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.113751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.113798 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.113880 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:49.114387 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57946: BlockingRecv error: recv got EOF from 127.0.0.1:57946 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 09:09:49.410195 22907 catalog_manager.cc:1259] Loaded cluster ID: 91fdc583a1004004a5f582be48786691
I20260620 09:09:49.410286 22907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: loading cluster ID for follower catalog manager: success
I20260620 09:09:49.411916 22907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: acquiring CA information for follower catalog manager: success
I20260620 09:09:49.413183 22907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e6ec637324d452982cb81e98b20c38c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:09:49.442528 22929 catalog_manager.cc:1259] Loaded cluster ID: 91fdc583a1004004a5f582be48786691
I20260620 09:09:49.442633 22929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: loading cluster ID for follower catalog manager: success
I20260620 09:09:49.444386 22929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: acquiring CA information for follower catalog manager: success
I20260620 09:09:49.445572 22929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30d8e8dfbc774d34857bc045bf20d791: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:09:49.633217 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:49.628717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.0.98.61:45953)
0620 09:09:49.628967 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.628975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.629098 (+ 123us) server_negotiation.cc:407] Connection header received
0620 09:09:49.629332 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.629335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.629507 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.629640 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:49.630007 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.630621 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.631498 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.631753 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.631836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:49.632227 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:49.632707 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:49.632819 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:49.632927 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 09:09:49.632985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260620 09:09:49.640748 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:49.636628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.98.60:37491)
0620 09:09:49.636869 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.636874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.636958 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:09:49.637224 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.637227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.637350 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.637482 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:49.637758 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.638299 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.639171 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.639408 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.639515 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:49.639852 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:49.640314 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:49.640459 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:49.640521 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 09:09:49.640583 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260620 09:09:49.648528 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:49.644035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.0.98.61:45953)
0620 09:09:49.644283 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.644287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.644558 (+ 271us) server_negotiation.cc:407] Connection header received
0620 09:09:49.644736 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.644739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.644793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.644875 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:49.645240 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.645982 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.647143 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.647498 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.647611 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:49.647938 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:49.648037 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:49.648104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:49.648291 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 09:09:49.648361 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 09:09:49.648927 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:49.643839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.98.60:37491)
0620 09:09:49.643946 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.643950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.644051 (+ 101us) server_negotiation.cc:407] Connection header received
0620 09:09:49.644220 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.644224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.644275 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.644358 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:49.644996 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.645816 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.647694 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.647977 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.648108 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:49.648363 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:49.648443 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:49.648507 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:49.648753 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 09:09:49.648808 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 09:09:49.649477 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:49.643702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.98.62:45877)
0620 09:09:49.643962 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:49.643967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:49.644373 (+ 406us) server_negotiation.cc:407] Connection header received
0620 09:09:49.644628 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:49.644631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:49.644810 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:49.644933 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:49.645977 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.646549 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.647741 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:49.648076 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:49.648176 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:49.648554 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:49.648963 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:49.649040 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:49.649232 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 09:09:49.649293 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260620 09:09:49.653160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22656
I20260620 09:09:49.660167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22731
I20260620 09:09:49.666929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 22805
2026-06-20T09:09:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3137 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 09:09:49 dist-test-slave-92t4 krb5kdc[23021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 09:09:49 dist-test-slave-92t4 krb5kdc[23021](info): set up 2 sockets
Jun 20 09:09:49 dist-test-slave-92t4 krb5kdc[23021](info): commencing operation
krb5kdc: starting...
W20260620 09:09:50.713495 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/test-user.keytab.
Jun 20 09:09:50 dist-test-slave-92t4 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T09:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T09:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 09:09:50.839694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 09:09:50.949863 23034 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:50.950155 23034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:50.950178 23034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:50.950217 23034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:50.954171 23034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:50.954224 23034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:50.954243 23034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:50.954259 23034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:50.954274 23034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:50.959414 23034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:50.960409 23034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:50.961606 23034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:50.970218 23042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:50.970176 23039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:50.970237 23034 server_base.cc:1034] running on GCE node
W20260620 09:09:50.970176 23040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:50.970957 23034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:50.972052 23034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:50.973244 23034 hybrid_clock.cc:648] HybridClock initialized: now 1781946590973221 us; error 26 us; skew 500 ppm
Jun 20 09:09:50 dist-test-slave-92t4 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:50.976203 23034 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 09:09:50.977175 23034 webserver.cc:460] Webserver started at http://127.0.98.62:41361/ using document root <none> and password file <none>
I20260620 09:09:50.977859 23034 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:50.977937 23034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:50.978168 23034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:50.980270 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data/instance:
uuid: "ba07eba17b9b40bfa8504d7b54c61e21"
format_stamp: "Formatted at 2026-06-20 09:09:50 on dist-test-slave-92t4"
server_key: "561ac88cc2cd37a14a0ab3d9b435597a"
server_key_iv: "3503267520c584438b8fdbcf818fb033"
server_key_version: "encryptionkey@0"
I20260620 09:09:50.980850 23034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba07eba17b9b40bfa8504d7b54c61e21"
format_stamp: "Formatted at 2026-06-20 09:09:50 on dist-test-slave-92t4"
server_key: "561ac88cc2cd37a14a0ab3d9b435597a"
server_key_iv: "3503267520c584438b8fdbcf818fb033"
server_key_version: "encryptionkey@0"
I20260620 09:09:50.984637 23034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 09:09:50.987202 23049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:50.988454 23034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 09:09:50.988592 23034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal
uuid: "ba07eba17b9b40bfa8504d7b54c61e21"
format_stamp: "Formatted at 2026-06-20 09:09:50 on dist-test-slave-92t4"
server_key: "561ac88cc2cd37a14a0ab3d9b435597a"
server_key_iv: "3503267520c584438b8fdbcf818fb033"
server_key_version: "encryptionkey@0"
I20260620 09:09:50.988693 23034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:51.022763 23034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:51.025806 23034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:51.026024 23034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:51.032968 23034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35627
I20260620 09:09:51.033280 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35627 every 8 connection(s)
I20260620 09:09:51.034037 23034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/data/info.pb
I20260620 09:09:51.036244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 23034
I20260620 09:09:51.036406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-0/wal/instance
I20260620 09:09:51.036754 392 external_mini_cluster.cc:1428] Setting key 7c30e2a6e8e71d8b602099f39e1f7350
I20260620 09:09:51.037029 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:51.040863 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:51.058162 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.043348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.0.98.62:35627)
0620 09:09:51.043921 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.043946 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.044175 (+ 229us) server_negotiation.cc:407] Connection header received
0620 09:09:51.045616 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.045621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.045827 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.045982 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.046897 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.047937 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.049442 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.050073 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.050261 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.053263 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.053269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.053280 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.053302 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.054987 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.055611 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.055617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.055618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.055724 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.056013 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.056015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.056016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.056487 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.056630 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.057033 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.057404 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":450,"spinlock_wait_cycles":55552,"thread_start_us":157,"threads_started":1}
I20260620 09:09:51.058169 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.043349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35627 (local address 127.0.0.1:51292)
0620 09:09:51.043981 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.044045 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.045220 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.046031 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.046036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.046397 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.046703 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.046717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.048073 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.048080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.049322 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.049332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.050239 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.050242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.050372 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.051057 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.051074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.053103 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.055107 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.055117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.055129 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.055465 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.055810 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.055812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.055814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.055927 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.056629 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.056635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.056768 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.056933 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.057299 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":43,"thread_start_us":146,"threads_started":1}
I20260620 09:09:51.060745 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:51.061584 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.061281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41586)
0620 09:09:51.061416 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.061498 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 09:09:51.061841 23050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
I20260620 09:09:51.062719 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.062534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41588)
0620 09:09:51.062630 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.062673 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 09:09:51.062892 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 09:09:51.064095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46655
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 09:09:51.112188 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } attempt: 1
I20260620 09:09:51.113183 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.112801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41590)
0620 09:09:51.112982 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.113090 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 09:09:51.113926 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.113701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41600)
0620 09:09:51.113812 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.113862 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 09:09:51.114117 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
W20260620 09:09:51.177088 23109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:51.177459 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:51.177497 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:51.177537 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 09:09:51.181573 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:51.181629 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:51.181648 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:51.181665 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:51.181681 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:51.182312 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } attempt: 2
I20260620 09:09:51.183148 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.182814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41604)
0620 09:09:51.182987 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.183059 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 09:09:51.183856 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.183639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41614)
0620 09:09:51.183745 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.183798 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 09:09:51.183997 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46655: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46655: connect: Connection refused (error 111)
I20260620 09:09:51.187454 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:51.188540 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:51.189630 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 09:09:51.197916 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:51.197963 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:51.197916 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:51.198329 23109 server_base.cc:1034] running on GCE node
I20260620 09:09:51.198913 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:51.199978 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:51.201151 23109 hybrid_clock.cc:648] HybridClock initialized: now 1781946591201139 us; error 24 us; skew 500 ppm
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:51.203874 23109 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 09:09:51.204748 23109 webserver.cc:460] Webserver started at http://127.0.98.61:35987/ using document root <none> and password file <none>
I20260620 09:09:51.205359 23109 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:51.205446 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:51.205669 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:51.207664 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data/instance:
uuid: "01d319e8c27841d5bc653fe3cb9ecf08"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "9ba2a6dd3969bc69e25400c3954bdcb7"
server_key_iv: "31da08dcd4305342ee7200db16486702"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.208218 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal/instance:
uuid: "01d319e8c27841d5bc653fe3cb9ecf08"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "9ba2a6dd3969bc69e25400c3954bdcb7"
server_key_iv: "31da08dcd4305342ee7200db16486702"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.211865 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 09:09:51.214318 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:51.215560 23109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:51.215705 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal
uuid: "01d319e8c27841d5bc653fe3cb9ecf08"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "9ba2a6dd3969bc69e25400c3954bdcb7"
server_key_iv: "31da08dcd4305342ee7200db16486702"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.215816 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:51.236442 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:51.239558 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:51.239779 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:51.247133 23109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46655
I20260620 09:09:51.247449 23176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46655 every 8 connection(s)
I20260620 09:09:51.248148 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/data/info.pb
I20260620 09:09:51.250811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 23109
I20260620 09:09:51.250958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-1/wal/instance
I20260620 09:09:51.251214 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:51.251302 392 external_mini_cluster.cc:1428] Setting key b1888cf713439643c87e2ae9bf61f69d
I20260620 09:09:51.255456 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:51.270090 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.257439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.98.62:35627)
0620 09:09:51.257620 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.257627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.258543 (+ 916us) server_negotiation.cc:407] Connection header received
0620 09:09:51.259750 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.259753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.259805 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.259881 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.261158 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.261761 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.262918 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.263294 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.263413 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.265943 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.265946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.265949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.265977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.267605 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.268227 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.268230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.268232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.268319 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.268738 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.268743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.268745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.268992 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.269190 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.269770 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.269909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 09:09:51.270947 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.257811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35627 (local address 127.0.0.1:51298)
0620 09:09:51.258339 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.258403 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.259554 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.260118 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.260125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.260459 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.260947 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.260960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.261881 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.261887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.262792 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.262801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.263467 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.263471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.263635 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.264241 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.264257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.265775 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.267752 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.267758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.267773 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.268082 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.268415 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.268418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.268421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.268620 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.269190 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.269197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.269486 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.269711 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.270060 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260620 09:09:51.272384 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:51.275159 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 09:09:51.278618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
/tmp/dist-test-taskU_KIjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41745
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:09:51.283996 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.272901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41630)
0620 09:09:51.273088 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.273103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.273182 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.273768 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.273771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.273945 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.274027 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.274033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.275111 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.275114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.275992 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.276000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.276736 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.276741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.276890 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.277518 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.277540 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.279488 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.281682 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.281685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.281688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.281910 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.282251 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.282257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.282262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.282374 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.282980 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.282984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.283394 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.283542 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.283722 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 09:09:51.285020 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.273092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.0.98.61:46655)
0620 09:09:51.273400 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.273410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.273430 (+ 20us) server_negotiation.cc:407] Connection header received
0620 09:09:51.273462 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.273465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.273645 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.273799 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.274156 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.274916 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.276094 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.276613 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.276724 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.279641 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.279651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.279673 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.279691 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.281568 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.282040 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.282046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.282048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.282142 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.282492 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.282498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.282501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.282850 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.283002 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.283641 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.283864 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260620 09:09:51.285735 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.275662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41640)
0620 09:09:51.275780 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.275794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.275868 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.276290 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.276293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.276493 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.276606 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.276612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.277709 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.277712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.278512 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.278523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.279143 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.279147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.279258 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.279770 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.279784 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.281371 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.284129 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.284132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.284134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.284366 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.284684 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.284689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.284692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.284809 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.285285 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.285288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.285388 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.285466 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.285593 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 09:09:51.286367 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.275720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.98.61:46655)
0620 09:09:51.276083 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.276089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.276101 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:09:51.276136 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.276138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.276168 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.276244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.276722 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.277589 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.278637 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.279024 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.279136 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.281541 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.281544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.281547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.281568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.284000 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.284499 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.284501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.284503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.284575 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.284918 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.284920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.284922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.285164 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.286042 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.286113 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.286221 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":303,"thread_start_us":202,"threads_started":1}
I20260620 09:09:51.287147 23177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:51.287243 23102 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:51.287895 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.287621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59576)
0620 09:09:51.287751 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.287808 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 09:09:51.288000 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.287636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59582)
0620 09:09:51.287838 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.287919 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 09:09:51.288311 23127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
I20260620 09:09:51.289095 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.288669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59594)
0620 09:09:51.288931 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.288989 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260620 09:09:51.289280 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.288953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59600)
0620 09:09:51.289153 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.289224 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 09:09:51.289319 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
W20260620 09:09:51.289422 23177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
I20260620 09:09:51.320521 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 1
I20260620 09:09:51.321509 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.321207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59610)
0620 09:09:51.321363 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.321429 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 09:09:51.322256 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.322047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59614)
0620 09:09:51.322156 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.322207 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 09:09:51.322376 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
I20260620 09:09:51.338678 23177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 1
I20260620 09:09:51.339556 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.339243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59624)
0620 09:09:51.339390 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.339469 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 09:09:51.340258 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.340054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59626)
0620 09:09:51.340162 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.340210 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 09:09:51.340368 23177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
W20260620 09:09:51.395598 23184 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 09:09:51.395924 23184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 09:09:51.396026 23184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 09:09:51.396127 23184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 09:09:51.397535 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 2
I20260620 09:09:51.398413 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.398119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59628)
0620 09:09:51.398270 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.398333 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 09:09:51.399139 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.398932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59632)
0620 09:09:51.399038 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.399089 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 09:09:51.399268 23102 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
W20260620 09:09:51.400858 23184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 09:09:51.400940 23184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 09:09:51.400986 23184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 09:09:51.401078 23184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 09:09:51.401118 23184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 09:09:51.406590 23184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35627,127.0.98.61:46655,127.0.98.60:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 08:13:29 UTC on ccycloud.quasar-srxcxo.root.comops.site
build id 4457054
I20260620 09:09:51.407725 23184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 09:09:51.408550 23177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 2
I20260620 09:09:51.408843 23184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 09:09:51.409381 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.409099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59644)
0620 09:09:51.409236 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.409304 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 09:09:51.410109 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.409877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59654)
0620 09:09:51.410006 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.410057 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 09:09:51.410240 23177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41745: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41745: connect: Connection refused (error 111)
W20260620 09:09:51.417541 23193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:51.417564 23191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 09:09:51.417514 23190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 09:09:51.417856 23184 server_base.cc:1034] running on GCE node
I20260620 09:09:51.418421 23184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 09:09:51.419581 23184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 09:09:51.420773 23184 hybrid_clock.cc:648] HybridClock initialized: now 1781946591420737 us; error 47 us; skew 500 ppm
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 09:09:51.423570 23184 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 09:09:51.424476 23184 webserver.cc:460] Webserver started at http://127.0.98.60:44407/ using document root <none> and password file <none>
I20260620 09:09:51.425109 23184 fs_manager.cc:359] Metadata directory not provided
I20260620 09:09:51.425200 23184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 09:09:51.425437 23184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 09:09:51.427506 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data/instance:
uuid: "e0d160c1897c4cc6888982542c95c3a9"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "f9f8f943780c95d9af769d30e7db7706"
server_key_iv: "cf8798077201a27c83f4e967cbebc054"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.428066 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal/instance:
uuid: "e0d160c1897c4cc6888982542c95c3a9"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "f9f8f943780c95d9af769d30e7db7706"
server_key_iv: "cf8798077201a27c83f4e967cbebc054"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.431819 23184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 09:09:51.434413 23200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 09:09:51.435670 23184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 09:09:51.435926 23184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal
uuid: "e0d160c1897c4cc6888982542c95c3a9"
format_stamp: "Formatted at 2026-06-20 09:09:51 on dist-test-slave-92t4"
server_key: "f9f8f943780c95d9af769d30e7db7706"
server_key_iv: "cf8798077201a27c83f4e967cbebc054"
server_key_version: "encryptionkey@0"
I20260620 09:09:51.436049 23184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 09:09:51.450366 23184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 09:09:51.453514 23184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 09:09:51.453743 23184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 09:09:51.461179 23184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41745
I20260620 09:09:51.461476 23252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41745 every 8 connection(s)
I20260620 09:09:51.462267 23184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/data/info.pb
I20260620 09:09:51.465157 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 09:09:51.465929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu as pid 23184
I20260620 09:09:51.466059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskU_KIjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781946309842113-392-0/minicluster-data/master-2/wal/instance
I20260620 09:09:51.466394 392 external_mini_cluster.cc:1428] Setting key d3d2d3695226bff3855cb71acdf15d2c
I20260620 09:09:51.469600 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:51.481727 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.471698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.98.62:35627)
0620 09:09:51.471822 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.471826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.472402 (+ 576us) server_negotiation.cc:407] Connection header received
0620 09:09:51.472682 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.472686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.472755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.472833 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.473691 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.474191 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.475184 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.475419 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.475533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.478064 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.478068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.478073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.478107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.479758 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.480224 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.480226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.480227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.480299 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.480775 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.480778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.480780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.481001 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.481149 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.481460 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.481577 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:51.481823 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.471742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35627 (local address 127.0.0.1:51314)
0620 09:09:51.472238 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.472320 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.472489 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.473009 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.473015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.473441 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.473558 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.473564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.474310 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.474314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.475058 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.475067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.475574 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.475577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.475656 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.476191 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.476210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.477912 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.479856 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.479866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.479872 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.480141 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.480426 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.480429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.480431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.480523 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.481146 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.481152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.481256 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.481369 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.481494 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
I20260620 09:09:51.481828 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.467706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.0.98.60:41745)
0620 09:09:51.468114 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.468129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.468164 (+ 35us) server_negotiation.cc:407] Connection header received
0620 09:09:51.468803 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.468819 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.469177 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.469455 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.470397 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.472015 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.473242 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.473913 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.474086 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.476617 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.476626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.476641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.476673 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.478723 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.479225 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.479230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.479234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.479332 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.479640 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.479642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.479644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.479992 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.480139 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.480498 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.480790 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260620 09:09:51.483767 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 09:09:51.493485 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.484546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.61:46655)
0620 09:09:51.484687 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.484691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.484704 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:09:51.484746 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.484749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.484795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.484866 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.485446 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.486237 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.487310 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.487678 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.487771 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.489972 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.489976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.489979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.490006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.491705 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.492145 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.492147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.492148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.492218 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.492534 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.492538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.492540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.492779 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.492862 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.493249 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.493364 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 09:09:51.493487 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.484269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46655 (local address 127.0.0.1:41650)
0620 09:09:51.484544 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.484558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.484637 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.484912 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.484917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.485122 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.485239 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.485247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.486356 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.486359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.487185 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.487195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.487784 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.487788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.487884 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.488418 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.488433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.489837 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.491806 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.491809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.491812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.492049 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.492304 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.492310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.492315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.492438 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.492898 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.492901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.493019 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.493170 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.493319 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260620 09:09:51.494275 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
I20260620 09:09:51.494467 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.484297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.98.62:35627)
0620 09:09:51.484403 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.484407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.484461 (+ 54us) server_negotiation.cc:407] Connection header received
0620 09:09:51.484567 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.484570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.484618 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.484691 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.485446 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.486233 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.487310 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.487656 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.487765 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.490551 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.490554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.490557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.490581 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.492277 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.492789 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.492792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.492793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.492922 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.493284 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.493290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.493292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.493533 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.493869 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.494223 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.494330 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 09:09:51.498445 23102 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 3
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 09:09:51.501374 23177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } attempt: 3
I20260620 09:09:51.503123 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.494775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59674)
0620 09:09:51.494883 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.494896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.494969 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.495233 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.495236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.495372 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.495484 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.495494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.496436 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.496440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.497023 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.497031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.497523 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.497526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.497596 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.498057 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.498070 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.499444 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.501643 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.501646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.501647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.501877 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.502153 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.502156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.502158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.502222 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.502568 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.502574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.502681 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.502793 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.502925 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 09:09:51.504138 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.494922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.98.60:41745)
0620 09:09:51.495047 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.495051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.495065 (+ 14us) server_negotiation.cc:407] Connection header received
0620 09:09:51.495099 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.495102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.495141 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.495228 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.495674 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.496329 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.497163 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.497412 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.497520 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.499583 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.499586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.499589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.499610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.501320 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.501984 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.501987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.501989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.502071 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.502299 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.502302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.502303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.502463 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.502562 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.503536 (+ 974us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.503988 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":48}
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:09:51.505106 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.495684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.98.61:46655)
0620 09:09:51.495788 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.495792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.495803 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:09:51.495894 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.495897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.495934 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.496016 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.496567 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.497012 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.497780 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.498004 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.498109 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.500950 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.500954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.500956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.500979 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.502785 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.503236 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.503241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.503243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.503338 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.503640 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.503644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.503646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.503865 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.504079 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.504743 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.504913 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":46}
Jun 20 09:09:51 dist-test-slave-92t4 krb5kdc[23021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781946591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 09:09:51.508285 23253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: Bootstrap starting.
I20260620 09:09:51.509123 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.499475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.0.98.60:41745)
0620 09:09:51.499867 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.499876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.499888 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:09:51.499925 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.499927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.499958 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.500564 (+ 606us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.500659 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.501512 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.502850 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.503325 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.503777 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.505615 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.505618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.505621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.505642 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.507367 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.507802 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.507807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.507809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.507891 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.508152 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.508157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.508159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.508341 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.508487 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.508762 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.508882 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260620 09:09:51.509558 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.498874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59678)
0620 09:09:51.499058 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.499071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.499140 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.500118 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.500123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.500309 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.500428 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.500435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.501644 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.501647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.502459 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.502469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.503447 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.503453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.503568 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.504019 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.504033 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.505467 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.507501 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.507504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.507506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.507703 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.507979 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.507982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.507985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.508075 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.508487 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.508491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.508589 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.508697 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.509110 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 09:09:51.511598 23253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:51.512226 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: Bootstrap starting.
I20260620 09:09:51.512272 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.501743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41745 (local address 127.0.0.1:59694)
0620 09:09:51.502283 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 09:09:51.502297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 09:09:51.502367 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 09:09:51.503104 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 09:09:51.503107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 09:09:51.503289 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 09:09:51.503410 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.503419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.504661 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.504664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.505897 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 09:09:51.505908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.506526 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.506531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 09:09:51.506629 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.507018 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 09:09:51.507034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 09:09:51.508346 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 09:09:51.510682 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.510685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.510688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.510926 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.511240 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 09:09:51.511243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 09:09:51.511245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 09:09:51.511334 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 09:09:51.511787 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 09:09:51.511791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 09:09:51.511903 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 09:09:51.511983 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 09:09:51.512121 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":1}
I20260620 09:09:51.512543 23253 log.cc:824] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:51.512887 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.502611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.98.60:41745)
0620 09:09:51.502882 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.502890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.502901 (+ 11us) server_negotiation.cc:407] Connection header received
0620 09:09:51.502926 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.502929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.502964 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.503519 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.503697 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.504527 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.506056 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.506361 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.506495 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.508495 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.508499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.508501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.508524 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.510555 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.511060 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.511062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.511064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.511140 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.511451 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.511456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.511461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.511666 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.512538 (+ 872us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.512618 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.512730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
I20260620 09:09:51.514951 23102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:51.515113 23253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: No bootstrap required, opened a new log
I20260620 09:09:51.515712 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: Bootstrap starting.
I20260620 09:09:51.515836 23102 log.cc:824] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:51.515923 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 09:09:51.517006 23076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ba07eba17b9b40bfa8504d7b54c61e21"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324
I20260620 09:09:51.518338 23177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: Neither blocks nor log segments found. Creating new log.
I20260620 09:09:51.519261 23177 log.cc:824] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: Log is configured to *not* fsync() on all Append() calls
I20260620 09:09:51.519869 23253 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.520170 23253 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d160c1897c4cc6888982542c95c3a9, State: Initialized, Role: FOLLOWER
I20260620 09:09:51.520579 23253 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.521097 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.521237 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.521509 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: No bootstrap required, opened a new log
I20260620 09:09:51.521960 23253 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:51.526063 23102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: No bootstrap required, opened a new log
I20260620 09:09:51.525969 23177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.526146 23177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d319e8c27841d5bc653fe3cb9ecf08, State: Initialized, Role: FOLLOWER
I20260620 09:09:51.526389 23177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.526762 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.526886 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.527585 23177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:51.525830 23184 master_runner.cc:186] Error getting master registration for 127.0.98.62:35627: OK, instance_id { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" instance_seqno: 1781946591031007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:09:51.528985 23184 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:09:51.528951 23102 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.529206 23102 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba07eba17b9b40bfa8504d7b54c61e21, State: Initialized, Role: FOLLOWER
I20260620 09:09:51.529489 23102 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.529934 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.530104 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.530730 23102 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 09:09:51.532814 23076 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 09:09:51.532927 23076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:09:51.533018 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:51.533129 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:51.534242 23076 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.531535 23109 master_runner.cc:186] Error getting master registration for 127.0.98.62:35627: OK, instance_id { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" instance_seqno: 1781946591031007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 09:09:51.534799 23109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 09:09:51.535148 23076 leader_election.cc:288] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [CANDIDATE]: Term 1 election: Requested vote from peers 01d319e8c27841d5bc653fe3cb9ecf08 (127.0.98.61:46655), e0d160c1897c4cc6888982542c95c3a9 (127.0.98.60:41745)
W20260620 09:09:51.536273 23299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:51.536413 23299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:51.536347 23151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "01d319e8c27841d5bc653fe3cb9ecf08"
I20260620 09:09:51.536583 23151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [term 0 FOLLOWER]: Advancing to term 1
W20260620 09:09:51.537074 23274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 09:09:51.537150 23274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 09:09:51.537314 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e0d160c1897c4cc6888982542c95c3a9"
I20260620 09:09:51.537529 23227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 09:09:51.538569 23227 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba07eba17b9b40bfa8504d7b54c61e21 in term 1.
I20260620 09:09:51.539050 23052 leader_election.cc:302] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba07eba17b9b40bfa8504d7b54c61e21, e0d160c1897c4cc6888982542c95c3a9; no voters:
I20260620 09:09:51.539358 23285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 09:09:51.539569 23285 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [term 1 LEADER]: Becoming Leader. State: Replica: ba07eba17b9b40bfa8504d7b54c61e21, State: Running, Role: LEADER
I20260620 09:09:51.539912 23285 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } }
I20260620 09:09:51.541791 23151 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba07eba17b9b40bfa8504d7b54c61e21 in term 1.
I20260620 09:09:51.542093 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba07eba17b9b40bfa8504d7b54c61e21. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.542188 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:51.551007 23308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 09:09:51.554056 23308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 09:09:51.555477 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.542928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.0.98.62:35627)
0620 09:09:51.543053 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.543058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.544877 (+ 1819us) server_negotiation.cc:407] Connection header received
0620 09:09:51.545755 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.545758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.545800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.545846 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.547667 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.548413 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.549550 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.549875 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.549941 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.551832 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.551836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.551839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.551866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.553613 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.554119 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.554124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.554128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.554212 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.554487 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.554490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.554492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.554713 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.554822 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.555183 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.555309 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 09:09:51.565559 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.543643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.0.98.60:41745)
0620 09:09:51.543774 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.543779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.545129 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 09:09:51.545515 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.545518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.545569 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.545620 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.549187 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.549941 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.554669 (+ 4728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.555030 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.555116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.560417 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.560422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.560425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.560455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.562384 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.562895 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.562899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.562900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.563006 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.563580 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.563585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.563590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.563784 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.563886 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.565232 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.565376 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 09:09:51.567003 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:51.544279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.0.98.61:46655)
0620 09:09:51.544383 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.544388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.556046 (+ 11658us) server_negotiation.cc:407] Connection header received
0620 09:09:51.556234 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.556237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.556291 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.556344 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.558914 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.559581 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.560344 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:51.560676 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:51.561137 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:51.563071 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 09:09:51.563077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 09:09:51.563080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 09:09:51.563108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 09:09:51.564864 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.565377 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.565380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.565382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.565460 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 09:09:51.566050 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 09:09:51.566056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 09:09:51.566060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 09:09:51.566270 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 09:09:51.566382 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:51.566721 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 09:09:51.566841 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 09:09:51.579171 23227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [term 1 FOLLOWER]: Refusing update from remote peer ba07eba17b9b40bfa8504d7b54c61e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:51.579171 23151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [term 1 FOLLOWER]: Refusing update from remote peer ba07eba17b9b40bfa8504d7b54c61e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 09:09:51.579927 23285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:51.580621 23303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 09:09:51.585083 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba07eba17b9b40bfa8504d7b54c61e21. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.585219 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.590375 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba07eba17b9b40bfa8504d7b54c61e21. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.590503 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.596155 23303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.596300 23303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:51.596422 23312 mvcc.cc:205] Tried to move back new op lower bound from 7298853239099424768 to 7298853238951206912. Current Snapshot: MvccSnapshot[applied={T|T < 7298853239099424768}]
I20260620 09:09:51.598045 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.598167 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.599009 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.599136 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21 [sys.catalog]: This master's current role is: LEADER
I20260620 09:09:51.601438 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba07eba17b9b40bfa8504d7b54c61e21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35627 } } peers { permanent_uuid: "01d319e8c27841d5bc653fe3cb9ecf08" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46655 } } peers { permanent_uuid: "e0d160c1897c4cc6888982542c95c3a9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41745 } } }
I20260620 09:09:51.601562 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 09:09:51.601759 23315 mvcc.cc:205] Tried to move back new op lower bound from 7298853239099424768 to 7298853238951206912. Current Snapshot: MvccSnapshot[applied={T|T < 7298853239099424768}]
I20260620 09:09:51.605595 23308 catalog_manager.cc:1347] Generated new cluster ID: f1d7ea9d29d04fb8a5e7be45bf469729
I20260620 09:09:51.605655 23308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 09:09:51.649649 23308 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 09:09:51.651044 23308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 09:09:51.676303 23308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba07eba17b9b40bfa8504d7b54c61e21: Generated new TSK 0
I20260620 09:09:51.677201 23308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 09:09:51.709753 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.706505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.0.98.62:35627)
0620 09:09:51.706643 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.706648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.707835 (+ 1187us) server_negotiation.cc:407] Connection header received
0620 09:09:51.708778 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.708782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.708839 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.708898 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.709614 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51350: BlockingRecv error: recv got EOF from 127.0.0.1:51350 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 09:09:51.711352 23259 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.707279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.0.98.60:41745)
0620 09:09:51.707415 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.707420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.710154 (+ 2734us) server_negotiation.cc:407] Connection header received
0620 09:09:51.710488 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.710491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.710540 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.710596 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.711202 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59718: BlockingRecv error: recv got EOF from 127.0.0.1:59718 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 09:09:51.712626 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.708154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.98.61:46655)
0620 09:09:51.708267 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.708273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.711711 (+ 3438us) server_negotiation.cc:407] Connection header received
0620 09:09:51.711883 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.711887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.711937 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.711987 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.712504 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41674: BlockingRecv error: recv got EOF from 127.0.0.1:41674 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 09:09:51.717675 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.715386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.98.61:46655)
0620 09:09:51.715491 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.715495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.716522 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 09:09:51.716773 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.716777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.716869 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.716932 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.717554 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41676: BlockingRecv error: recv got EOF from 127.0.0.1:41676 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 09:09:51.722851 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.721371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.98.61:46655)
0620 09:09:51.721474 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.721477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.721938 (+ 461us) server_negotiation.cc:407] Connection header received
0620 09:09:51.722150 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.722153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.722204 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.722262 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.722753 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41678: BlockingRecv error: recv got EOF from 127.0.0.1:41678 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 09:09:51.728861 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.727229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.98.61:46655)
0620 09:09:51.727336 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.727340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.727813 (+ 473us) server_negotiation.cc:407] Connection header received
0620 09:09:51.728071 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.728074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.728125 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.728179 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.728740 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41694: BlockingRecv error: recv got EOF from 127.0.0.1:41694 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 09:09:51.737550 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.735653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.98.61:46655)
0620 09:09:51.735809 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.735813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.736387 (+ 574us) server_negotiation.cc:407] Connection header received
0620 09:09:51.736622 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.736625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.736686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.736746 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.737428 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41700: BlockingRecv error: recv got EOF from 127.0.0.1:41700 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 09:09:51.750000 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.748313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.98.61:46655)
0620 09:09:51.748434 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.748449 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.748973 (+ 524us) server_negotiation.cc:407] Connection header received
0620 09:09:51.749236 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.749239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.749291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.749342 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.749876 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41710: BlockingRecv error: recv got EOF from 127.0.0.1:41710 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 09:09:51.770233 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.768919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.0.98.61:46655)
0620 09:09:51.769114 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.769119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.769284 (+ 165us) server_negotiation.cc:407] Connection header received
0620 09:09:51.769469 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.769472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.769531 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.769633 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.770111 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41712: BlockingRecv error: recv got EOF from 127.0.0.1:41712 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 09:09:51.805696 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.804342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.98.61:46655)
0620 09:09:51.804565 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.804570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.804724 (+ 154us) server_negotiation.cc:407] Connection header received
0620 09:09:51.804905 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.804915 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.804963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.805104 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.805579 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41716: BlockingRecv error: recv got EOF from 127.0.0.1:41716 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260620 09:09:51.872920 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:51.871871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.0.98.61:46655)
0620 09:09:51.872039 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:51.872043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:51.872093 (+ 50us) server_negotiation.cc:407] Connection header received
0620 09:09:51.872222 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:51.872225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:51.872272 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:51.872345 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:51.872794 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41732: BlockingRecv error: recv got EOF from 127.0.0.1:41732 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 09:09:52.004251 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:52.003051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.0.98.61:46655)
0620 09:09:52.003212 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.003216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.003268 (+ 52us) server_negotiation.cc:407] Connection header received
0620 09:09:52.003394 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.003397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.003457 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.003565 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:52.004087 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41748: BlockingRecv error: recv got EOF from 127.0.0.1:41748 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 09:09:52.263314 23182 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 09:09:52.262298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.98.61:46655)
0620 09:09:52.262424 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.262428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.262512 (+ 84us) server_negotiation.cc:407] Connection header received
0620 09:09:52.262631 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.262633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.262679 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.262760 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 09:09:52.263178 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41750: BlockingRecv error: recv got EOF from 127.0.0.1:41750 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 09:09:52.534426 23287 catalog_manager.cc:1259] Loaded cluster ID: f1d7ea9d29d04fb8a5e7be45bf469729
I20260620 09:09:52.534518 23287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: loading cluster ID for follower catalog manager: success
I20260620 09:09:52.536244 23287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: acquiring CA information for follower catalog manager: success
I20260620 09:09:52.537658 23287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01d319e8c27841d5bc653fe3cb9ecf08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:09:52.538374 23274 catalog_manager.cc:1259] Loaded cluster ID: f1d7ea9d29d04fb8a5e7be45bf469729
I20260620 09:09:52.538434 23274 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: loading cluster ID for follower catalog manager: success
I20260620 09:09:52.540076 23274 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: acquiring CA information for follower catalog manager: success
I20260620 09:09:52.541175 23274 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0d160c1897c4cc6888982542c95c3a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 09:09:52.781522 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:52.777639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.61:46655)
0620 09:09:52.777859 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.777863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.777933 (+ 70us) server_negotiation.cc:407] Connection header received
0620 09:09:52.778105 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.778108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.778233 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.778328 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:52.778591 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.779111 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.779967 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.780206 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.780288 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:52.780552 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:52.781000 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:52.781222 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:52.781267 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 09:09:52.781337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260620 09:09:52.787743 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:52.784411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.98.60:41745)
0620 09:09:52.784662 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.784666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.784679 (+ 13us) server_negotiation.cc:407] Connection header received
0620 09:09:52.784758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.784760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.784872 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.784969 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:52.785168 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.785688 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.786456 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.786670 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.786743 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:52.786985 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:52.787429 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:52.787521 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:52.787569 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 09:09:52.787618 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260620 09:09:52.794775 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:52.790669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.98.61:46655)
0620 09:09:52.790799 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.790809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.791120 (+ 311us) server_negotiation.cc:407] Connection header received
0620 09:09:52.791520 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.791526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.791588 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.791672 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:52.792046 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.792785 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.793731 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.793965 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.794081 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:52.794329 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:52.794398 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:52.794473 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:52.794563 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 09:09:52.794616 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 09:09:52.795600 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:52.791060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.0.98.60:41745)
0620 09:09:52.791241 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.791246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.791258 (+ 12us) server_negotiation.cc:407] Connection header received
0620 09:09:52.791383 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.791386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.791461 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.791538 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:52.791776 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.792578 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.794147 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.794475 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.794551 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:52.795109 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:52.795185 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:52.795274 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:52.795388 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 09:09:52.795442 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 09:09:52.796103 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 09:09:52.790619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.98.62:35627)
0620 09:09:52.790874 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 09:09:52.790879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 09:09:52.791184 (+ 305us) server_negotiation.cc:407] Connection header received
0620 09:09:52.791270 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 09:09:52.791273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 09:09:52.791439 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 09:09:52.791551 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 09:09:52.791913 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.792490 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.794144 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 09:09:52.794487 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 09:09:52.794555 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 09:09:52.795163 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 09:09:52.795524 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 09:09:52.795617 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 09:09:52.795922 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 09:09:52.795986 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":63,"threads_started":1}
I20260620 09:09:52.799094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 23034
I20260620 09:09:52.805603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 23109
I20260620 09:09:52.811561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskU_KIjO/build/debug/bin/kudu with pid 23184
2026-06-20T09:09:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3144 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6282 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282961 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST